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ABSTRACT 

This study reports the results of an evaluation of 
the operational feasibility of a new model for training personnel for 
educational research and development. Also, recommendations are made 
for future testing of the concept on a larger scale. The evaluation 
attempts to identify practical aspects of the pilot program which 
could be modified, maintained, or improved. A total of 14 key 
questions were investigated in such areas as achievement by 
apprentices, apprentice reaction to the program experience, the host 
institution's reactions, the degree to which the training program and 
experience could be exported, and suggestions for other types of 
models. Although a number of model modifications were recommended, it 
is concluded that the pilot training program was successful and 
effective. In addition, it is felt that variations of the model 
developed can be of use in independent training programs as an 
adjunct to regular academic programs or for inservice training. The 
inservice model is recommended as the most immediately viable for 
educational research and development training efforts. (Author) 
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ABSTRACT 



The goal of this project is to evaluate the operational feasibility of a new 
model for training personnel for educational research and development, with 
recommendations for future testing of the concept on a larger scale. The focus 
of the evaluation is to identify practical aspectsof the pilot program which 
could be modified, maintained, or improved, as well as recommend new directions 
for future efforts. 

Fourteen key questions were investigated. They included achievement by the 
apprentices, apprentice reaction to the program experience, the reactions of 
the host institution, the degree to which the training program and experience 
could be exported, and suggestions for other types of models. 

While a number of modifications of this model were recommended, the pilot 
training program is considered successful and effective, in addition, vari- 
ations of the model developedcan be of use in independent training programs, 
as an adjunct to regular academic programs, or for in-service training. The 
in-service model is recommended as the most immediately viable for educational 
research and development training efforts. 
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CHAPTER 1 



INTRODUCTION 



Background Discussion 



A decade or so ago educational research and development were carried out 
mainly in graduate schools. The main theme was basic research. This basic 
research was conducted by dedicated scholars working individually. They 
drew on students who became their assistants. Much of what these assist- 
ants learned about research and development, they learned from their mentors. 

As these efforts grew in scale and became more formalized, graduate courses 
and programs slowly evolved. The advent of federal support to formal 
training programs and the needs of the burgeoning educational R&D industry 
in laboratories and centers hastened this evolution. 

Today, however, the field of educational research and development has become 
considerably broader than the basic research of decades past. The skills are 
different, the settings have shifted from the academic mold of the univer- 
sity to the worlds of production, dissemination, and practical impact. 

There is no doubt a melding of pragmatic and theoretical/academic activities, 
but there has been little formalization of programs to serve both masters 
at once . The gap has been recognized, and we again observe the need for 
apprentices. The more traditional training institutions cannot fully meet 
that need. Since the bulk of educational research and development is shift- 
ing away from the university, the university is no longer the only viable 
training ground. The "action" is moving to the marketplace, much of the 
learning is by "doing," and many of the exportable ideas for education 
come from agencies whose very existence depends on addressing problems 
with practicality and impact. Basic research knowledge is still necessary, 
but no longer sufficient. There is a new ethic of accountability that 
traditional programs have rarely faced. 

Strangely enough, the old paradigm of the mentor/apprentice reappears in 
an informal way at Educational Laboratories, R&D Centers, and at private 
research agencies. The "mentors" teach their "apprentices" (junior staff), 
albeit in an informal way. It would seem logical that if the field of 
educational research and development requires the varied expertise of 
academic background combined with practical application, training should be 
designed to accommodate both. There is no better way to satisfy the desirabil- 
ity of applied efforts in training, given an academic base, than to partici- 
pate in an on-going, established, accountability-oriented activity. Most 
university graduate programs cannot, or do not, satisfy this need. 

Educational research is a much broader world of work than ever before. It 
has come a long way from writing a workbook, or studying a single facet of 
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learning, writing it up, and publishing it in a journal. In turn, profession- 
al educators currently employed in educational R&D are no longer involved in 
only pure educational research as in years past. There is now a much greater 
emphasis on development, dissemination, and impact. The real world of 
educational research and development requires some knowledge of such diverse 
topics as finance and budgeting, day-to-day production engineering and 
monitoring, economics and sources of support, vehicles for dissemination, 
marketing, long-range planning, manpower needs identification, communica- 
tions, and presentation techniques. These are the real life facets of 
current educational research and development, and an apprenticeship would 
serve as the vehicle by which an individual could become acquainted with 
such realities. 

Schools, as they traditionally exist, do not provide many of these experiences. 
It is unrealistic to set up traditional training programs for the "Renais- 
sance Man" in educational R&D. The basic goal then becomes the means by 
which the most realistic, efficient, and effective exposure to the field is 
provided. Apprenticeships currently provide the most realistic approach 
to fulfilling this need. The apprenticeship notion is extremely logical 
and flexible. With the current need for diversity in manpower, and the 
current absence of reality-based programs to meet those diverse needs, the 
development of various approaches to training educational research and de- 
velopment personnel becomes a critical task. 



Perspectives for the 70' s 



Obviously, reality-based activities take place in agencies and institu- 
tions which are actually involved in educational research and development. 

A major consideration, however, is not so much where reality-based activities 
should take place, but rather, how and by whom these facilities might be 
used. 

There are three major categories of persons who might participate in a 
reality-based (or apprenticeship) experience in educational research and 
development. First, there are those persons who have considerable work 
experience, even in research and development, but who have had no experience 
in educational research and development. Educational R&D facilities might 
be used to provide entry training to those already possessing basic R&D 
skills, with an eye toward re-directing those skills and experiences to the 
field of education. This is defined as "cross-training," and serves as the 
basic description for the AIR training model being evaluated in this study. 
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A second category of persons would include those who are university under- 
graduate or graduate students in educational research. Traditionally, 

R&D facilities have been used to supplement this formal academic train- 
ing in basic educational theory or educational research. In many instances, 
universities attempt to provide their own "reality-based" programs, often 
labeled "internships." These internships may be internal or external to 
the university. There are a variety of disadvantages, however, to univer- 
sity-based internship programs which are discussed more fully in Chapter 
4. Prime among these disadvantages, though, is that the scope of experiences 
necessary for a realistic, work-oriented program is quite restricted in most 
instances. Most universities simply do not have access to the scope of pro- 
ject resources necessary to an effective internship or apprenticeship, nor 
can they effectively engineer realistic work experiences when those experi- 
ences must be correlated with the academic calendar. 

The third category of persons who might be trained in an apprenticeship type 
program includes those who are already employed in educational research and 
development, but who wish to develop their skills further. With this pop- 
ulation, the main purpose of an apprenticeship would be intensive, specific, 
in-service skill development, rather than broad, entry-level training in 
educational R&D, as in the first two cases. 

All three of these types of populations need to be considered in light of 
current and projected manpower needs in educational R&D. To cross-train 
or supplement traditional academic training (which includes the first two 
categories of persons), implies a new influx of manpower . These are persons 
who are not new in the field, but who could become so after an apprenticeship 
experience. In view of the immediate prospects in educational R&D employ- 
ment, this problem is probably not as great now as it was only two or three 
years ago. 

The third category of potential apprentices includes those who are already 
in the field of educational R&D. There would be no new influx of manpower, 
but rather an upgrading of existing manpower . In this latter case, the main 
issue is that of improving the existing skill repertoire, rather than creat- 
ing a repertoire. 

In view of the likelihood of the creation of large scale institutional 
development in the form of NIE, and in view of the long range stability of 
Educational Laboratories, Centers, and non-profit institutions such as AIR, 
and ETS, this latter category is likely to be the most important of the 
three, at least in the immediately foreseeable future. 

It is indeed, a very strong point for discussion as to whether "new bodies" 
or "better bodies" are needed in educational R&D today. As recently as a 
year or two ago, a small number of studies indicated that there would be 
personnel shortages in the years to come. These estimated shortages ranged 
from severe to modest. It may well be that current economic setbacks and 
spotty commitments to educational R&D make the more severe predicted shortages 
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somewhat suspect. However, the more conservative estimates are still no 
doubt viable, even given an uncertain outlook for educational R&D. There 
will be manpower needs, but all would agree that the short-range demand is 
in a period of slack. 

What then for the immediate future? Apprenticeships are acknowledged to 
be desirable and necessary to the field, but the current issue is to whom 
those apprenticeships might be directed. In the case of cross-training, 
we have an influx of new manpower from other disciplines, most notably 
the physical sciences. As regards models for training housed in universities, 
there is no doubt that existing R&D facilities can benefit such programs. 

But, the immediate future calls for a need to make educational R&D workers 
better, not just simply to make more of them. These are times of accounta- 
bility, and "tight money." Educational R&D then, needs to make the best of 
what it already has on hand. This is not to say that apprenticeship train- 
ing for new personnel should be abandoned; it is only to say that in-service 
type programs might provide the best new effort to serve educational R&D 
during the 1970's. While the 50' s and 60' s were more concerned with man- 
power shortages , the 70' s should be more concerned with existing manpower 
development . 

The program being evaluated in this study was conducted as if there were 
moderate national manpower shortages. When the program was conceived a 
year ago, there was the feeling that educational R&D would benefit from the 
infusion of new perspectives, represented by experienced personnel from 
physical science R&D. While it is still believed that this would be the 
case, in retrospect from this spring of 1972, the audience was perhaps not 
the highest priority, given an analysis of current and immediate manpower 
needs. However, the program itself, was based on a very solid concept. 
Relatively short-term, reality-based apprenticeship programs very much need 
to be developed. The task in this program was to develop a new model for 
apprenticeship training in a real-life setting. While the model employed 
a population being cross trained, it would be equally appropriate for the 
in-service staff development with already employed trainees. The audience 
for training and the intent of training might vary, but the apprenticeship 
is still a vital supplement to educational R&D personnel development. 
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The Purpose of the Evaluation Study 



In September of 1971 AIR undertook the development of a new model for 
the training of personnel in educational research and development. The im- 
petus for this effort was based in the long felt need for programmed experi- 
ences in educational R&D which would supplement varieties of academic train- 
ing with more reality-based activities. The purpose of this study is to 
explore the operational feasibility of that model , with recommendations for 
future testing of the concept on a larqer scale. 

The original intent of the training program was a pilot effort to develop 
a model. While the program had design and structure, it was intended to 
be preliminary and small-scale. The purpose of this evaluation study is 
to identify and "iron-out" the initial kinks in the development of the model 
and conduct of the program. In the formal sense, this study should be 
considered formative evaluation for future efforts, rather than a summative 
evaluation of the apprenticeship model. 

Therefore, while an examination of student outcomes is certainly considered 
in this evaluation, it is by no means the only focus of the evaluation. 

The evaluation looks at the model as a whole. The basic inquiry is of the 
practicality of the model, from a variety of viewpoints. These viewpoints 
include not only what the apprentices learned, but how they reacted to the 
experience, the reactions of the host institution, the degree to which this 
training program and experience were unique to the setting, the difficulties 
in program operation, the apparent demand for such a program, the ease 
with which jobs were obtained by the apprentices, and so forth. 

At the center of this evaluation is what we could learn by conducting the 
program. We were mainly interested in exploring whether or not this was a 
viable model. The evaluation is intended to describe and explain what happen- 
ed; point out those facets which worked well and those which could be 
changed; and, in general, contribute to minimization of potential pitfalls, 
should the model be used in subsequent training efforts. 



The Model to be Evaluated 



The mode 1 and training experiences are discussed in considerable detail 
in the companion final report. Development of an Apprenticeship Work- 
Study Program Model for the Cross-Training of Physical Scientists for 
Work in Educational Research and Development (brant Nck OEG-O-71-4712) . 

However, to summarize, the setting was a private, non-profit research corpora- 
tion (AIR-Palo Alto); the number of trainees was six; the length of the 
pilot test was three months; the training/education relied on a combination 
of lecture, independent study, and apprenticeship. Tor the most part, 
instiuction was individualized; no university credit was offered. The 
"students" were mature workers in the process of changing vocations— 
namely, from research and development in the physical sciences to research 
and development in the behavioral sciences (i.e., education). The 
qeneral idea was to see if an educated and experienced group, which was, 
however, naive about educational research and development, could receive 
enough training in a relatively short period of time to make them effective 
workers, attractive to potential employers in educational R&D. 

The "students" were six unemployed aerospace professionals. The fact that 
these were unemployed physical scientists had little crucial bearing on 
the development of the model. The fact that they were all physical 
scientists had a great deal to do with the model. The model was developed 
not to be a substitute or formal training but rather a complement. There- 
fore, the main problem with such a group was not one of training, per se, 
but rather the transfer of old skills to a new area--"cross-t.raining," as 
it were, rather than training or retraining. 

While at AIR, the students participated in lectures and seminars in key 
areas of education; independently studied areas related to either these 
key topics, topics of interest, or topics related to work activities; 
and participated in apprenticeship activities under the guidance of senior 
and project AIR staff. At least 50% of the time during the 13-week training 
program was devoted to apprenticeship activities. In most cases, indepen- 
dent study and apprenticeship activities were unique to the student. 

Students were expected to study and work over the course of the normal AIR 
work-day of 8:30 AM to 5:00 PM, and in many instances, beyond those hours 
when topics or activities demanded special attention. For their efforts, 
they received a stipend of $75 per week plus $15 per week per dependent to 
cover living and relocation expenses. 

In short, the basic idea was to conduct a relatively short-term, work- 
oriented apprenticeship program for unemployed aerospace personnel to 
train them for work in educational research and development as an alternative 
exploration to existing and traditional programs. 
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CHAPTER 2 



THE NATURE OF THE EVALUATION 



Organization of the Evaluation 



Although the model has four conceptual components (recruitment/selection, 
instructional, apprenticeship, and job placement), they are somewhat 
restrictive as the sole structure for the organization of evaluation. It 
is true that certain issues for exploration are based in a single component, 
but there are questions of interest which cut across components, such as 
trainee reaction to the general program experience. In order then, that 
the assessment draw comprehensively from all appropriate data sources and be 
conceptually flexible, the design focused on a series of key questions. 

They are: 

A. How effective is the program in terms of knowledge and 
skills gained? 

B. How successful is the program? 

C. How general izable or exportable is the model? 

D. What is the trainee reaction to the program experience? 

E. What should be the content and procedures for follow-up? 

F. How responsive is the manpower pool? 

G. What procedures should be followed for the selection of trainees? 

H. What are the implications to the host institution in conducting 
an apprenticeship training program? 

I. What are effective procedures for the monitoring of trainee 
progress? 

J. What is the role of traditional tests in trainee selection 
and assessment? 

K. Who applied? What were their characteristics? 

L. What were reasons for applicant rejection and were they viable 
in retrospect? 

M. In addition to how might trainees be selected, who might be 
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selected in future replications of the model? 

N. What modifications of this model, or other types of models, 
might be considered in future efforts? 

There are, then, fourteen questions and issues of interest to be explored 
in this evaluation. The first thirteen are discussed in Chapter 3, and 
the last question is discussed separately in Chapter 4. 



Data Sources and Instrumentation 



While a trainee sample of only six would be too small for a comprehensive 
evaluation of a training model, it is sufficient to entertain questions of 
the operational feasibility of a model. As such, it is highly desirable 
to do so prior to large-scale model implementation and evaluation. 

Data gathering devices included an analysis of telephone responses, a 
standardized test, an application form., a writing sample, personal inter- 
views with trainees and AIR staff, survey questionnaires, a weekly chronicle 
of activities and reactions, self-ratings, staff observations and critical 
incidents, behavioral records and unobtrusive measures, and staff ratings. 

In order to provide a comprehensive picture of when and how the data were 
gathered to address the various questions of interest, the sources employed 
for each question will be presented separately in this overview: 

A. How effective is the program in terms of knowledge and skills gained? 

A variety of sources were drawn upon. These included: 

(1) A trainee analysis of the degree to which each was able to meet the 
various objectives of the objectives bank proved to be highly in- 
formative. The format of the working instruments which each trainee 
used was designed to account for those objectives which were attainable 
before the program, those which were enhanced by the program, and 
those which could now be accomplished only because of the program. 

In addition, the objectives are clustered into eleven topical areas. 

A sample of this instrument is offered in Appendix A of the companion 
final report on the training program itself. In addition, a summary 
of those responses is offered in Section A of Chapter 3 of this report. 



(2) A pre- and post-standardized instrument in educational areas was also 
employed. The instrument selected was the Undergraduate Record 
Examination in Education published by Educational Testing Service. 

It is a two-hour test which deals with broad areas of educational 
problems. Trainees were administered the test as part of the 
selection procedure prior to the start of the program, and underwent 
a readministration of the test during the last week of the training 
program. While statistical pre, post-analysis could not be performed 
on the group as a whole, a description of the individual change scores 
was a point of interest. 

(3) In a final survey questionnaire, trainees were *sked to specify those 
areas in which particular growth had been noted, and, in addition, 

to offer an overview statement about the degree to which the training 
program had generally enhanced their knowledge of educational research 
and development. A sample of that questionnaire is given in Appendix 
A of this report. 

(4) Throughout the program, supervisory staff were queried as to the 
static or increasing effectiveness of apprentices within their 
apprenticeship activities. These were not tendered in a formal docu- 
ment, such as a questionnaire, but consisted of irregularly spaced 
interviews with senior staff, the content and timing of which depended 
on the apprentice, what was being done in the apprenticeship, and how 
long it had been in process. 

(5) Throughout the program, the immediate staff of the project had close 
contact and working relationships with the trainees. The observations 
of staff, the noting of critical incidents, the summaries of trainee 
behavioral records, and an awareness of assorted unobtrusive measures 
all contributed to evidence of trainee growth or lack thereof. These 
included the degree to which there were quicker understandings of prob 
lems and tasks with a reduction of questions for guidance from staff, 
greater initiatives in task accomplishment, an awareness of sources 

of information, and the general ability to discuss and make contribu- 
tion to problem solution in educational topics. This evidence did not 
take the form of ratings in the strict sense, but were verbally codi- 
fied at weekly staff meetings. 



B . How successful is the program? 

The word "successful 11 has two related meanings here. The first is the 
degree to which the program produced "graduates" who are attractive to 
potential research and development employers; the second criterion, which 
is in the long run perhaps the most important, is whether the trainees 
obtain employment. Sources for this issue are fairly straight-forward. 
They include: 



(1) inquiries from potential employers who were aware of the program's 
existence; 

(2) the extent to which trainees gained contacts and received inter- 
views from potential employers; 

(3) the degree to which AIR was interested in hiring one or more of 
the trainees; and 

(4) whether or not the apprentices ultimately secured employment. 



C. How general izable or exportable is the model? 

This question is not based in any one type of data source or answerable 
through any of the particular data gathering devices thus far employed. 

It can be easily understood that this is an issue based on judgment and 
total experience. The simple question is whether or not this training 
program could be done in another setting. There are many factors of an 
environmental or geographic/demographic nature which were explored. 

These included the location of AIR in Palo Alto, the availability of 
materials at this particular site, the variety of apprenticeship activities 
offered by AIR and the surrounding community, and manpower availability. 



D. What is trainee reaction to the program experience ? 

Data sources included: 

(1) a diary manintained daily and submitted weekly by trainees „ 

These "logs" included trainee reactions to activities, program 
content, staff relations, apprenticeships, and general program 
experiences; 

(2) the final survey questionnaire which queried trainees in one of 
its sections as to their summary judgments of the training program 
experience; 

(3) informal comments and conversations with trainees which were 

noted throughout the program and summarized in the program monitor's 
program development summary. 



E. What should be the content and procedures for follow-up? 

Again, the answer to this question is a matter of accumulated judgment. 
However, one approach was developed and utilized which was particularly 
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well-suited to a small sample of trainees. T !iis was: 

(1) a two-month follow-up telephone survey. 

While the medium of the telephone was well-suited to six people, the struc- 
ture of the questions would generalize to any number of persons being followed 
up. The questions asked in that survey are given in this section. For larger 
samples, a mailed questionnaire would have been appropriate. 



F. How responsive is the manpower pool ? 

The main data source was: 

(1) an analysis of the density of telephone inquiries in response to 
initial public notification of the program, and a companion analy- 
sis of the number of inquirers who followed up with actual sub- 
mission of the application form. 

G . What procedures should be followed for the se l oction of trainees ? 

This question must again be ultimately based on experience. However, a 
reference point is offered by the techniques employed in the program. 

They included: 

(1) an application form; 

(2) a standardized test on educational problems; 

(3) a writing sample; 

(4) a personal interview; and 

(5) multi-stage staff ratings. 

H. What are the implications to the host institution in conducting an 
apprenticeship training program? 

Main sources of information on this isnue included: 

(1) interviews with AIR staff who were involved with apprenticeship 
and seminars; 

(2) interviews with senior AIR administrative staff; 



(3) summary reactions of immediate program staff. 



I . What are effective procedures for the monitoring of trainee progress ? 

Again, the ultimate answer lies in accumulated judgments, but reference points 
have been provided by: 

(1) weekly or bi-weekly conferences between each trainee and the 
program monitor; 

(2) the system of daily log-keeping by each trainee; 

(3) a mid-program and final-program self assessment by each trainee 
of objectives attained; and 

(4) staff observation. 



J . What is the role of traditional tests in trainee selection and assess- 
ment ? 

Data for this question were sourced in: 

(1) the judged effectiveness of the twice-administered standardized 
test, and 

(2) the judged appropriateness of alternative instruments which were 
examined for potential application. 



K. Who applied? What were their characteristics? 
The sole data source was: 

(1) selected data from the application form. 



L. What were reasons for applicant rejection and were they viable in 
retrospect? 

This question was explored through: 

(1) records maintained during the selection process; and 

(2) judgments of the selection committee gathered through informal 
interviews. 



M. In addition to how trainees might be selected, who might be selected 
in future replications of the model? 

No one data source was appropriate to answer this question. The issue is 
speculative and is based on discussions with staff about the characteris- 
tics of people who might have had the potential to successfully adapt to 
such a program. 



N • What modifications of this model, or other types of models, might be 
considered in future efforts ? 

The data base for this question obviously draws comprehensively from all 
the preceding questions and their data sources. No one source can be 
particularly identified. The results are not really answer;, but rather 
suggestions and possibilities given one type of total program experience 
in one model. 

It is important to note that in this and other questions of its type, the 
data base is experiential and the suggested conclusions and recommendations 
make no pretense as being the only or final answers. They are offered as 
preliminary findings based on the accumulated, and hopefully sound, judgments 
of experienced professionals in educational research and development. 



CHAPTER 3 



EVALUATION OUTCOMES 



As noted in Chapter 2, each issue for evaluation is presented in separate 
sections of this chapter. The thirteen questions are identified by sections 
A through M. The body of each section contains the related general findings, 
and a discussion which includes comments , conclusions, and/or recommendations. 



Findings Based on Objectives Mastery 

The most definitive information on this issue is found in a summary of 
self-ratings by each trainee on the objectives bank which was used to 
define the overall curriculum of the program. A system of self-assessment 
was employed as the only realistic substitute to actually testing each 
trainee on each objective. The development of that instrumentation was 
not feasible given the limits of the program. However, self-assessments 
were modified or supported by staff observations, where possible. 

The bank consists of 130 objectives, which are clustered into eleven areas. 
The number of objectives in each area is as follows: 



Trainees were asked to indicate which of the objectives could be met before 
the program, which were acquired only during the program, which could be 
performed before the program but were supplemented by the program, and which 
objectives could not yet be performed at the time of inquiry. In addition, 
of those objectives which could be performed, trainees were asked to indicate 
their degree of expertise for each, i.e., slight, moderate, or strong. 

This instrument was administered on two different occasions. When it was 
administered at the mid-point of the program, further information was ob- 
tained as to which of the objectives each trainee would like to emphasize 




Library Skills 

Instrumentation- — 

Data Collection — 

Data Analysis — 

Proposal Preparation — 

Product Development 

Evaluation — — — - 

Marketing, Dissemination, Implementation 

Write-up and Publication 

Scheduling/Management — 

Apprenticeship Experiences 



17 

10 

21 

10 

10 

13 

15 

10 

5 

8 

12 



in the balance of the program. The main purpose of the mid-point administra- 
tion was to assist the program monitor in assessing progress to date, and to 
provide guidelines for future learning activities. The instrument was re- 
administered at the end of the program as a summary effort. That final 
administration formed the basis for the discussion to follow. 

While each and every one of the 130 objectives will not be discussed here, 
the eleven topic areas can be explored as separate groups. A number of 
summary statements are evident. 



*0n the average, there was no one category where partici- 
pation in the program was the sole reason for all the 
trainees now being able to fulfill all the objectives. 

This would obviously not be expected for a variety of reasons. Primary among 
these reasons is that the trainees were selected because they were already 
able to perform certain skills relevant to educational R&D when they entered 
the program. The picture might have been different had the sample been 
totally naive. 



*Looking at only the skills which can now be performed, 
it is interesting to note that in eight of the eleven 
topical areas, the number of trainees who felt that the 
program was a) either the sole source of skill develop- 
ment or b) supplemented development of previous skills, 
exceeded the number who felt that they could perform 
the skills in the area just as well before the program. 

In order of magnitude, these topical areas on which the program had greatest 
impact for the group as a whole were: 

- Data Anlaysis 

- Proposal Preparation 

- Instrumentation 

- Product Development 

- Evaluation 

- Library Skills 

- Data Collection 

- Marketing, Dissemination, Implementation 

It is not in the least surprising that the impact of the program was most 
highly felt in the areas of Data Analysis, Proposal Preparation, and Instru- 
mentation. As part of guided, independent study activity, the trainees 
worked through modules on data analysis which were being developed by AIR 
for the Far West Laboratory, and were in the final draft stage. These 



exercises provided substantive knowledge in data analysis, as well as experi- 
ence in product review and refinement. 

In regard to proposal preparation, four of the six trainees had direct contact 
with either the writing or production of proposals. The fact that the program 
influenced skills in this area is also not surprising. In addition, pro- 
posal writing is a common task at private research houses such as AIR, as 
well as bing a recurring topic of conversation, A trainee who spends 
three months in such a milieau learns a great deal by watching and listen- 
ing, to say nothing of actually participating. 

The area of Instrumentation was enhanced due to the kinds of activities 
which are also common to AIR type organizations. Within the variety of 
apprenticeship positions, all trainees had contact with either the develop- 
ment or utilization of forms, questionnaires, data summary sheets, manuals, 
and the like. These are common to many research and development activities. 



*The program seemed least effective in the areas of 
Write-up and Publication, Data Collection, and Mar- 
keting, Dissemination, and Implementation. 

This was due in part, to a lack of immediate interest on the part of the 
trainees, and in general, to the fairly short time span of the program; 
these are activities for which long term involvement and commitment are 
likely to be characteristic. 



*When the program is looked at as the sole source for 
objectives attainment, the areas to which the greatest 
average number of trainees responded that they could 
not have performed certain of the objectives unless 
they had been in the program were: 

- Library Skills 

- Product Development 

- Evaluation. 

While these areas were not as strongly developed as some others by the end of 
the program, on the other hand, there was not a comparable base of expertise 
at the beginning of the program. However, growth was experienced, and 
perhaps, given the starting points, the most relative growth took place in 
these three areas. 



*The areas in which the trainees felt they were 
strongest before the program started were: 

- Scheduling/Management 

- Apprenticeship Objectives 

- Proposal Preparation 

- Data Analysis 



*Those areas of the program which were most active in 
in enhancing previously acquired skills were: 

- Proposal Preparation 

- Apprenticeship Objectives 

- Scheduling and Management 

This is to say that of those skills which could be performed before the pro- 
gram, additional inputs were made to the greatest degree in these three 
areas. Thus, the combination of background expertise and additional training 
makes these areas among the strongest in the trainees' repertoires. 

At the end of the program, it can be summarized that trainees, as a group, 
ranked their strengths in the eleven topical areas as: 

(1) Apprenticeship Objectives 

(2) Data Analysis 

(3) Proposal Preparation 

(4) Scheduling/Management 

(5) Instrumentation 

(6) Library Skills 

(7) Product Development 

(8) Evaluation 

(9) Write-up and Publication 

(10) Data Collection 

(11) Marketing, Dissemination, Implementation 

Thus, in summary, it can be said that the training program was strongest in the 
reinforcement and transfer of existing skills. In addition, growth was made 
in areas which were relatively new to them in an educational context such as 
Library Skills, Product Development, and Evaluation. There were three areas 
in which the trainees had relatively little past experience and on which the 
program made relatively little impact. It is likely that growth in these 
areas would also have been experienced given greater interest and longer 
program tenure. 

On the immediate pages are presented the individual responses of the trainees 
to this objectives survey (Table 1). The numbers in the boxes indicate the 
number of trainees who responded to the various stems for each objective. 
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To develop, under supervision, draft 
materials. 



Findings Based on Test Results 



A second contributor to the exploration of this first evaluation question 
was the ETS te r t in educational areas. The issue of the role of current 
standardized testing in programs such as this is briefly discussed in 
Section I of this chapter, but suffice to say here that initial expectations 
were that pre- and post-program score comparisons would not offer very 
interesting results. 

In general, the thinking was that the specific items on the test did not, 
for the most part, reflect program activities. Therefore, if very little 
score increments were noted, it would have been neither surprising nor dis- 
appointing. However the test, initially used in the selection process, was 
readministered, nonetheless, as a point of interest. 

The test was heavily laced with topics such as educational history, note- 
worthy educational theorists, classroom management techniques, and learning 
theory. None of these topics was stressed to any degree in the program, 
save, perhaps, learning theory. However, for reasons including assigned 
readings and informal discussions, trainees picked up knowledge in these 
areas to a surprising degree. Individual test score gains of the six 
trainees ranged from 5% to 43%, with an average gain of 19%. 

Considering that trainees were in the program only three months, and that 
the staff made absolutely no attempt to "teach to the test" or even review 
it after the initial administration, these gains are considered quite 
remarkable, even considering measurement error. One trainee commented 
that his increment did not seem to be so much due to any particular directed 
study, assigned readings, and the like, as it was to just being in an en- 
vironment of educational research and development and developing a some- 
what different way of thinking about education. Both trainees and staff 
were pleasantly surprised at these outcomes. 



Findings Based on Trainee Survey 

A third source of information on this question was the previous^’ mentioned 
trainee survey completed during the last days of the program. The- first 
three questions of the survey are of particular interest here: 

(1) To what extent do you feel you know more about educational 
R&D in general, than you did thirteen weeks ago? 

(2) In which specific topic areas did you gain the most information 
from the program? 

(3) What specific skills related to educational activities were 
additionally or solely developed because of your participation 
in the program? 
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Trainees responded to these questions in the form of open-ended essays. For 
the sake of clarity and integrity of reproduction, some key phrases from 
those responses are directly quoted below for each of the three questions. 



Trainee Responses 

(1) To what extent do you feel you know more about educational R&D 
in general, than you did thirteen weeks ago? 

"... I gained most information about the balance 
between educational research efforts and developmental 
efforts. In my opinion, funds spent on development 
far outweigh those spent on research although the 
creation of more firm knowledge of research is needed 
before more development is justified." 

"I have become acquainted with some of the products 
and systems developed such as CAI, programmed materials 
teaching modules, etc., and feel almost like I have 
met B. F. Skinner." 

"Thirteen weeks ago I knew what the average citizen 
knows about educational r&d which is to say that I 
was passively aware somewhere some activity was 
probably taking place, other than in the universities. 

Now I know where much of this activity is taking place, 
what types of institutions and people are involved, 
the sources of funding that supports this research, 
and something about the general nature of this research 
and the direction it is taking." 

(2) In which specific topic areas did you gain the most information 
from the program? 

"... Skinnerian analysis of behavior, instructional 
objectives, programmed learning, the use of incentives, 
statistical procedures applicable to educational 
research, observing and interviewing, the psychology of 
development." 

"... proposals, instructional materials." 

"My greatest gain was in learning about the specific 
areas in which AIR employees are funded and are current- 
ly working." 



(3) What specific skills related to educational activities were 
additionally or solely developed because of your participation 
in the program? 

"The thirteen weeks was a most welcome opportunity 
to think deeply about education. This process 
included, I believe, some increase in my ability 
to identify problems (by reading and discussion), 
analyze them, and record the results in writing. 

I know of no skills solely developed because of 
the program." 

"I feel that skill development was rather minimal, as 
the areas ... in which I have had no experience 
were not obviously offered in the program." 

"... the management of research data, the use 
of statistics, interviewing, unobtrusive measures 
of behavior, use of ERIC, the development of 
questionnaires, general knowledge skills in the 
areas of educational psychology, funding sources 
and proposals, educational products (knowledge 
and developmental), the use of incentives and 
reinforcement, and the specific skills needed 
for the development of training courses and 
modules." 

"... the skill of locating the right information 
source for a particular problem." 

"... writing proposals and instructional materials, 
research - library skills." 

These responses obviously reflect varying degrees of intensity and detail. 
The trainees varied considerably as to their expressiveness and the degree 
to which they perceived the program to be providing new experiences and 
opportunities in various specific areas. Each response was quite different, 
which no doubt reflects the individual differences of the trainees, as well 
as the intended, individualized nature of the program. However, it is 
judged that common to most of these responses is a degree of sophistication 
about educational research and development which was not likely to have 
been exhibited prior to the start of the program. 



Findings from Staff 

Finally, in terms of staff opinion, the trainees made effective contribu- 
tions to ongoing tasks, and in two or three cases, outstandingly so. 
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Apprenticeship supervisors generally felt that the nature of assigned tasks 
were quickly grasped, and that the performance of those tasks was more than 
adequate. By way of specific citations of knowledge and skills demonstra- 
ted, one trainee wrote very well, had some long-fomenting ideas on the edu- 
cational process, and took the program opportunity to combine these in the 
production of two different position papers related to ongoing projects 
at AIR. Another trainee who had previously been very naive about library 
research, very quickly learned about the major documents and search 
processes in education and conducted a thorough literature review as part 
of an apprenticeship activity. In two other instances, trainees made strong 
inputs in the development of data collection instruments and a manual. 

In general, the staff felt that there were no particular skills or know- 
ledges which were outstandingly common to all of the trainees, save the 
general ability to easily handle routine tasks as would be the case with 
any educated, experienced worker. It was the opinion that trainees 
performed differentially well in different areas when tasks called for 
more sophisticated activity. In these instances, these abilities were called 
upon in the various apprenticeships. However, as indicated, trainees were 
also given the opportunity to learn and practice skills previously foreign 
to them, such as educational literature reviews and instrument design. 

These skills appeared to be adequately developed within the program, but 
called for additional monitoring and guidance on the part of the staff. 



Discussion 



It is first obvious to note that each trainee emphasized different skill 
areas, and that furthermore the intensity with which each was dealt varied 
considerably from trainee to trainee. It is not possible to cite any one 
skill or knowledge area which was equally attended to or practiced or 
developed by all trainees. This was certainly one outcome of having indi- 
vidualized the program. However, as a general statement, it is certain 
that the program had considerable impact . 

Every trainee was far more capable of functioning in an educational R&D 
environment at the end of the program as compared to the beginning of the 
program. There is strong evidence for this in the initiatives exhibited 
in conducting and completing tasks, in the sophistication of questions 
and conversations concerning education, in the awareness of materials and 
sources of information, in the adaptation to the work style and goals of 
an educational R&D agency, and finally, in the growing confidence of 
staff that all the trainees were approaching the point where they could 
dependably and intelligently make inputs to tasks usually reserved for 
more experienced workers. 

Given any educated and industrious worker, not enough can be said for 
a broad range of experiences in educational R&D. The seminars, inde- 
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pendent study, and discussions were all valuable in providing focus 
and perspective, an understanding of the setting, and a knowledge of 
some of the more esoteric vocabulary. But the apprenticeship itself 
was felt to be of greatest value, at least from a staff point of 
view, for it appeared to be closest to the heart and intent of the model, 
offered the most day-to-day reality of what it means to work in education- 
al R&D, and provided the strongest foundation for the development of 
effective workers. 

The trainees apparently did not quite share this view. Question #5 
of the final survey asked which of the various types of activities seemed 
to be mostuseful in the acquisition of skills and/or information, i.e., 
seminars, independent readings and contacts, informal discussions, appren- 
ticeship activities. While most mentioned the apprenticeship as being 
particularly valuable, not one listed it first as the outstanding compo- 
nent. The top choice was split between seminars and independent readings. 

Perhaps the disparity lies in the question, for it did not specify whether 
it meant skills and knowledge in education or skills and knowledge in 
educational R&D work . These different views on the question might suggest 
differences in views on the program itself between staff and trainees. 

Perhaps the reality-based, work-oriented nature of the program should have 
been stressed more, or verbalized more often. It is suspected that trainees 
viewed the program as more academic than the cross-training of their skills 
to be applied in day-to-day work in a different setting. 

This does not necessarily mean that the model needs to be different, but 
rather that the communication of the importance of the work portion of the 
model be clearer. This issue is basically one of understanding the purposes 
of the various components of the model: as indicated, it did not seem 

to make any apparent difference in the quality of apprenticeship activities 
of each trainee. 

From a trainee viewpoint, the most overt gains of the program seemed to 
be generally in the areas of orientation to educational R&D and termin- 
ology. After those initial foundations, they seemed able to draw on their 
own resources. This was desirable, for the program was intended to be one of 
transition. Trainees were expected to bring something to the program, 
and they were very much in accord that what they all had the most of was 
general problem solving skills. The program to them was the opportunity to 
reapply these skills in a new setting with a slightly different texture 
to the problems. 

To try to get the trainees to be consistently more specific about what 
these skills were was an extraordinarily difficult task which was resisted. 
While their expression was accomplished at various levels, these skills were 
rarely expressed at very specific levels. In terms of finite explication, 
this effort was, in the end, not as specifically comprehensive as had been 
hoped. However, the purpose of doing this in the first place was to 
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identify trainee strong points and special interests, in order that they 
might be reinforced in studies and apprenticeships. This was accomplish- 
ed, but the process was not as easily defined, regulated, or predictable 
as originally had been thought. 

The problems in this may not have lain solely with the trainees relative 
inability to easily express their specific skills, for it could probably 
be equally admitted that the field of educational R&D has a similar prob- 
lem. While a great deal of progress has and is being made, there is not, 
as yet, an agreed-upon, comprehensive, specific skills bank relative to 
educational R&D; there have simply been few systematic explications of 
the myriad of tasks involved in RDD&E work. In addition, there have been 
few efforts, if any, to formalize assessment criteria. The program did 
employ a sizable pool of knowledge and skills objectives which were based 
on recent efforts, such as those of the Far West Lab, and also based on 
in-house experience. But, it is most difficult to feel secure about 
its comprehensiveness. 

Therefore, when trainees find it difficult to express what they can do, 
and when the field of educational R&D has not codified itself well 
enough to easily explicate what it does in order to provide cues to 
trainees, it is easy to see that communication is less than crisp. It 
is, then, highly recommended, that efforts toward this codification continue 
in earnest, for it is basic to calculated and directed personnel training 
in educational research and development . There also need to be additional 
efforts directed toward the development of assessment instruments, given 
task and skill identification. 

There is a related issue in the degree to which academic type studies in 
education are related to actual educational R&D work. Personnel train- 
ing in this area, and in this model, was mainly designed to produce effec- 
tive workers. The fact that they might also be knowledgeable scholars is 
additionally important, but secondary. There have been few efforts to 
separate knowledge in education from skills in educational R&D work . 

They are not necessarily the same, and, in many instances, would not have a 
functional relationship. However, it is easy to think that such would be 
the case, for it is the natural inclination of the basically academically- 
oriented field of educational R&D personnel. In this program, for example, 
in spite of efforts to the contrary, we initially tended to fall into the 
trap of treating the trainees more pedagogical ly than pragmatically. It 
quickly became apparent and we quickly reminded ourselves that we were 
not there to entertain trainees with academia, but to do every-day work- 
and to share those experiences with the trainees. Academics had to be 
covered where necessary, but we frequently needed to remind ourselves 
that this was not our reason for being. There is no ready resolution, 
but frequent reminders to both staff and trainees that apprenticeship pro- 
grams emphasize work more than basic academics might contribute to a 
better focusing of common goals. 
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In summary, it is evident from the results that knowledge and skills in 
education and knowledge and skills in educational R&D work were indeed 
absorbed by the trainees. Much of this effort on the part of staff was 
calculated and it worked with each trainee to varying degrees. There were 
some hesitant steps in not having a comprehensive educational R&D objective 
bank, in not having assessment instruments, and in occasionally shifting 
emphasis from work to academia. But, the first two will be solved with 
future efforts in these directions, and the third is a matter of constant 
awareness on the parts of both staff and trainees. 

Given the length of the program, it must be termed a success in the 
acquistion of knowledge and skills. There is no reason to believe that 
such would not be the result on a larger scale. 



B. How successful is the program? 



As mentioned in Chapter 2, the word "successful" had two originally intended 
meanings. The first had to do with whether the program produced "graduates" 
who would be attractive to this and other educational research and develop- 
ment organizations. This would include such factors as exhibiting familiar- 
ity with the field, having had known successful experiences in productive 
activities related to educational RAD, generally having credibility and 
inspiring confidence, and as a result, be of interest to potential em- 
ployers. 

The second meaning of the word "successful," could in some sense be con- 
sidered the ultimate criterion of program effectiveness, and that was 
whether or not trainees secured employment. 



Findings in terms of potential employability 

All of the trainees would be attractive to many or all educational agencies 
in selected capacities. Three of the trainees would have been particularly 
attractive to AIR. Had we had the activities and manpower shortages to 
justify the act, they would have been gladly hired on a permanent basis. 
Respectively, their strengths were writing and educational philosophy, 
creative inputs to educational products, and a good knowledge of statistics, 
experimental design, and basic psychology. 

AIR was certainly not the only group interested in a "finished" trainee. 

Three unsolicited contacts were received. The first occurred only three 
weeks after the start of the program. AIR was host to the Eighth Invitational 
Conference on Systems Under Construction for Career Education and Develop- 
ment at which the trainees were guests. The almost 100 participants rep- 
resented the United States Office of Education, certain of the Regional 
Educational Laboratories and Research and Development Centers, private indus- 
try such as ETS and, of course, AIR, individual school districts, the military 
and a variety of colleges and universities. In the opening ceremonies of 
the Conference, all six trainees were personally introduced and the purpose 
of their presence at AIR briefly explained. Out of this contact came one 
request for resumes and, in general, more information on the trainees. 

The contact diu not come to fruition for any of the trainees mainly due to 
timing. The program had many weeks yet to go, and the inquirer's staff 
needs were immediate. They were not as interested in a three week "graduate" 
as they were in a finished product. 

The second unsolicited contact was by one of the Regional Laboratories. 

As part of the apprenticeship; one of the trainees had participated in the 
thinking, writing, and production of a series of papers on the Employer 
Based Career Education Model. AIR was under contract to the Far West Lab 
at the time for this task. Because of this association, the trainee was 
asked to submit his resume to FWL, and was promptly hired by the Lab, 
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on a temporary basis, even before the training program was complete. 

During the employment, solid and satisfactory inputs were made to the 
task at FWL. 

The third inquiry came from one of the local community colleges who had 
been a cooperating partner in apprenticeship positions. As part of the 
college's overall planning and funding promotion activities, it was seeking 
one staff member to serve as Research Analyst. Two trainees were in- 
volved in these negotiations and competed with a number of applicants who 
had had no contact with the training program. The outcome was that one of 
the trainees was hired for the job. 



Findings in terms of actual employment 

Given that there was interest in the trainees, the second issue of actually 
obtaining jobs in now addressed. 

As indicated, one trainee found employment as a research analyst at a 
local community college (De Anza). The specific nature of the job calls 
for an analysis of the college's various educational programs in order to 
determine their relevance and cost-benefit to the community. 

A second trainee became Coordinator of the Work Experience Program at 
Gavilan College in Gilroy, California. This program deals with the secure- 
ment and certification for credit for work experiences of students which 
are outside the formal academic program. Both of these post-program jobs 
exceed $12,000 annual salary. 

A third trainee was employed on a part-time basis by AIR to continue work 
he had begun during the apprenticeship portion of the training program. 
While funds do not now permit regular employment, he is on-call for work 
on a contractual basis when needed. 

The fourth trainee was recalled by his previous employer, Boeing. While 
the new job is not identical to the old, it consists of a reapplication of 
his existing skills. While the trainee was keenly interested in education, 
and the new job holds no more security than the last, he could hardly turn 
down "a bird in the hand" after having been unemployed for some time. 

There were also two additional job offers to these employed trainees. 

One was tendered during temporary employment offering the possibility of 
permanent employment, contingent upon receipt of a contract. However, the 
contract did not materialize. 

A second offer was made to another trainee to administer psychological 
tests to patients at a psychiatric clinic in Florida. The trainee had 
not as yet made a decision at the time of this report. 



Therefore, counting the temporary employment of one trainee during the 
training program, six temporary or permanent job offers were potentially 
or actually realized for four of the trainees, and five of these six 
offers were directly involved with some phase of educational R&D activity. 



Discussion 



The initial contacts were highly encouraging as to whether or not trainees 
might be attractive to potential employers. Agencies in no way seemed to 
be "put off" or discouraged from considering the trainees for their staffs. 
This is in spite of the fact that the program was relatively short and 
that the backgrounds of the trainees were in areas not directly related 
to education. There appeared to be no particular problem in joining the 
trainees with other more logical personnel in the pools of applicants. 

This is no doubt related to the perceived needs of educational R&D 
agencies to have diverse competencies on their staffs. 

AIR interest in the trainees was quite strong. Perhaps it is not fair 

to compare this interest to those interests of groups who had not had direct 

contact with the trainees: we had the advantage of knowing the skills and 

competencies of each trainee through three months of experience. Per- 
haps the key point here is that even after that, some of these personnel 
became more attractive than ever as potential employees. Given some work 
experiences and familiarity with educational R&D, it grew more difficult 
to tell the trainees from regular junior staff. In these terms, the program 
does indeed work. 

This association with AIR seemed to be highly important to the trainees 
to a much greater extent than was ever previously imagined or predicted 

by the staff. There seemed to be some unspoken dependence on AIR for credi- 

bility, if not for actual employment. There were occasional comments 
that mentioning AIR within the community was met with recognition, and this 
strengthened the trainees's position. There were additional comments in 
a half-serious, half-jesting manner, that if AIR got some new contracts 
which required additional staff, they would be employed at AIR. It is 
very difficult to tell just how strong this reliance was, but there can 
be no doubt that it existed to some degree. There seemed to be a very 
subtle attitude that if AIR did not give them a job, AIR would certainly 
find one for them. This attitude seemed to exist in spite of repeated 
and candid comments by the staff that we could rot directly provide or 
obtain jobs for the trainees after the program terminated. 

Future programs need to be very wary of supporting or fostering perceptions 
of this type. No educational organization is or can be an employment 
agency. While the program provided and suggested contacts, assisted in 
resume revision, and gave general advice, in no way could we "get" 
someone a job. This was mentioned during recruitment and selection and 
was mentioned frequently during the program, but somehow it seems that it 



was never quite believed or understood equally well by all the trainees. 

In the long run, employment comes down to their own skills, their own 
contacts, and their own presentations of themselves. AIR could be and 
was a catalyst, but could never by the sole agent. 

As a result, not all of the trainees worked equally hard at job-seeking. 

Some were very industrious indeed, but others delayed and were fairly 
unsystematic, hoping that either something would break at AIR, or that 
AIR would provide some break, somewhere, for them. Those who worked hard- 
est at it were the ones who got the jobs. Those who seemed to rely more 
heavily on AIR did nut get jobs. These activities are discussed in detail 
in the companion final report on the training program itself ( 0EG-0-71 - 
4712). 

In addition, two of the trainees were most unwilling to relocate from 
their established home areas — one from the Bay Ares ana one from Seattle. 
This conflicted with statements of agreement to relocate offered at the 
time of admission to the program. While this is not unreasonable, it is 
unrealistic. Both areas are pockets of generally high unemployment. 

Seattle is depressed in all its economic sectors, and new employment in 
educational research and development is highly subdued. While the Bay 
Area is rich in activities of this sort, it is equally rich in qualified 
unemployed who also do not wish to relocate. There are jobs and there 
is a certain amount of the „sual turnover, but it is by no means a 
"buyer's market." Employment is not easy to find and secure. 

It was hoped that all trainees would enthusiastically go wherever there 
were jobs. As indicated, this was asked on the application form, and 
all were willing to relocate. But when it came down to actually being 
faced with relocation, such was not the case for at least two of the 
trainees. They were fairly lukewarm about even applying for jobs in more 
distant locations such as Ohio, and even relatively nearoy locations. 

Three trainees would not follow up leads in Sacramento, and two made little 
effort to follow up leads at the Far West Lab in Berkeley. This occurred 
even with the provision for paid travel for which these unemployed were 
eligible. 

In addition, one trainee refused to seriously entertain the typical entry 
level salary ranges which were associated with these job possibilities. 

This, too, was known would be the case before the program started, but 
the actual realities of lowered salaries apparently were not genuinely 
faced. If these trainees had been more flexible, their chances for employ- 
ment would have been considerably enhanced. 

On the other hand, half of the trainees were quite flexible and vigorous- 
ly pQrsued the strong job leads. One traveled as far as Florida and another 
as far as Wisconsin for interviews and follow up to their applications. 

However, it is important to mention that the training program was con- 
ducted during a period of extreme economic slowdown on a national scale. 
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Jobs were not easy to get in any area, save perhaps medicine, certain 
service areas, and the like. The generally tight economic situation 
during this time was definitely a factor, in both an optimistic outlook 
on the part of trainees, and the ease with which job contacts and 
possibilities were established. 

However, even given this, half the group is employed on a permanent or 
part-time basis in areas related to education, and all of the group 
were potentially attractive to educational research and development 
employers. 

With a batting average at or near 50%, the success of the program is 
considered great , even given its modest goals. It is almost impossible 
to state whether or not that success rate would have been maintained 
had there been 75 or 100 trainees at various agencies throughout the 
country. There are many factors of time and place, national economy, 
and individual attractiveness and flexibility. However, with its 
small sample, the program was a success in terms of its basic criteria. 



C. How generalizable or exportab le is the model ? 

Discussion 



There really are no "results" to this question, in the ordinary sense: it 

is a matter of observation and discussion. The basic issue is whether or 
not the program could be conducted equally well with replication at another 
site. The various components of the model are examined as to their trans- 
fer utility. There were four basic components: recruitment and selection, 

the instructional system, the apprenticeship, and job placement activities. 



Recruitment and selection component 

In this particular program, recruitment and selection were directed toward 
unemployed aerospace professionals from Seattle and the Bay Area. The 
specific procedures are discussed in greater detail in Sections F and G 
of this chapter. However, in general they were fairly traditional, were 
mostly done by mail, and did not involve a great deal of transportation 
and general logistics. 

More personal contact was required toward the end of this process, and the 
Bay Area group was, of course, able to come directly to AIR for final 
screening. In the final phases of the Seattle screening, one staff member 
traveled to Seattle for two days for test administration and interviewing. 
The location of AIR in Palo Alto had, or could have, very little influence 
on this process. 

There is no reason why much of what was done could not have been based any- 
where in the country and directed toward any audience in the country . Ads 
could be placed in distant papers (as was done), a local contact could be 
established (as was done), information and application forms could be 
exchanged by mail (as was done), and one or two staff members could travel 
to distant sites to conduct the final review and selection processes (as 
was done). There was nothing unique about AIR's staff or location which 
would make this process the slightest bit difficult to conduct at other 
sites. The only change would be that of slight increments in expenses: 
should the sites be varied and at great distances, expenditures for 
travel and communication would accordingly exceed those of this program. 
However, these in no way would interfere with the process of conducting 
this component of the program. 



The instructional component 

The instructional and apprenticeship components were certainly the most 
unique of the four, but not to the extent that it would be impossible to 
replicate them at another site. 
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The instructional component was dependent mainly on local staff and existing 
materials. Local staff expertise is one aspect of the entire program which 
could conceivably be situation-spec ific. Some of the lectures/seminars 
were of a nature which would make them conductable at any research and 
development agency. They included such topics as an overview of educational 
research and development, the general skills involved in educational R&D, 
and proposal development,, These should be no problem at any site. How- 
ever, some other of the topics were more academically specific, and re- 
flected the experiences, expertise, and academic backgrounds of staff members. 
Examples of these are behavioral objectives, developing instructional 
materials, and incentives and accountability. Locating staff personnel 
who are cognizant in these areas at any conceivable site might be a problem 
area in the model, for their ready-availability, and even existence, is 
not predictable. As it turned out, AIR had such personnel in its staff 
repertoire who were willing to participate in the program. However, 
while all these conditions might not prevail everywhere, it is strongly 
suspected that most major agencies would also have these general and 
specific capabilities among their personnel. 

As a side issue, but an important one, any agency involved in RDD&E 
training should have these diverse capabilities. If apprenticeship train- 
ing is intended to expose neophytes to the broad field of educational R&D, 
then the host institutions should have the capacity to reflect this range 
through its staff and activities. Among the more eligible agencies would 
be the larger, private research houses, federal and state educational 
agencies involved in education, the Regional Laboratories, and, probably, 
the Educational R&D Centers. Less effective training grounds would be new 
and small research agencies and specific departments of universities, for 
they are less likely to have a broad range of in-house competencies. 

Therefore, while AIR had a great deal to offer in the way of depth and 
scope of expertise, it is not the only site in the country. It could well 
be expected that the larger, older institutions of its type in the country 
could provide this as well. Thus, on this particular aspect, the model is 
likely to be exportable and replicatable, given the obvious variations of 
individual competencies to be found at any one site. _ 




Materials 



The materials used in the instructional component were not nearly as situation- 
specific, although their identification and availability could have some- 
thing to do with the capacities of AIR. Firstly, the identification of 
the best or most noteworthy materials in a field could easily be accomplished 
in-house. The immediate project staff was well aware of such fields as 
learning theory, child development, programmed instruction, interviewing 
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and counseling, statistics, measurement, and research design. If we 
needed to confirm materials in these areas, or identify materials in 
areas with which we were not as familiar, a staff member who could was 
nearby. Again, any major R&D agency, whose staff is professionally sound, 
should also have this capability. 

The materials which are referred to here are, as indicated, the leading, 
noteworthy materials in any one field. These are most likely to be in the 
form of established textbooks. While these and other materials seemed 
relatively sufficient at the time, the staff additionally felt the need 
for some ready, comprehensive source on all materials related to the topics 
of interest, as well as some index of their difficulty or the intended, 
appropriate audience. These materials would include, not only textbooks, 
but programmed texts, games and exercises, and workbooks. What was keenly 
wished for was some ready-access materials identification bank, much like 
ERIC, but stratified differently and with more user information. Such 
a bank is not yet available, but its institu tion is high ly recommended . 

Its aid to those who conduct training programs would be tremendous, both 
in terms of a time-saving device, and as confirmation that the materials 
used are appropriate and comprehensive. 

Any conscientious program staff wants to use the best, and not necessarily 
what is on the shelf. This identification was far more time-consuming and 
uncertain than had been imagined during the design of the program. In 
addition, throughout the program, we constantly heard of new efforts which 
sounded like they might be useful, but information on these efforts was 
spotty. We were not quite sure where to find out more about them, how 
much time they would involve for the trainees, where to get them, how 
much they would cost, or how quickly they could be obtained. It would 
save this and other training staffs of any type a great deal of effort 
and decision-making time if this information were readily available in 
a public, systematic way. 

In the availability of materials, AIR had no problems, once they were 
identified. We found easy access to all materials which were noted as 
being crucial or desirable for the curriculum. The ease with which these 
materials were obtained might be a weak spot in the model, but certainly 
not something which cannot be accommodated at other sites. AIR is 
fortunate in having a sizable in-house library of 1,600 books; 5,300 
research reports; and subscriptions to more than 350 journals. In addition, 
it maintains two full-time librarians to serve staff research needs. 
Obviously, not every agency has this capability in the very same building. 

To supplement this already rich source, the AIR library maintains an 
official, professional affiliation with all Stanford Libraries. If 
a reference is not stored at AIR, it is readily available from Stanford. 

In the sense of accessibility of materials, AIR has particular advantages. 

However, nothing was unique about the materials themselves. They are 
well known and nationally available. The implications for the replica- 
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tion of the model are that other host institutions need to a) have or be 
near major, comprehensive libraries which are likely to be rich in educa- 
tion-related documents, or b) prior to the start of any training program , 
stock up on documents, journals, books and so forth which It thinks Tt 
might utTlize . AIR's situation allowed for almost spontaneous retrieval 
of materials. Agencies which do not have this capability could certainly 
get the same materials, but it would have to be a more studied effort, 
allowing considerable lead-time. 

There is one additional point on materials. We did not have six copies 
of all documents, and sometimes they were needed. Should a program be 
even larger in scale, it might be difficult to get dozens of copies of 
everything. Even in this day of Xeroxing, it is not feasible to make 
multiple copies of a whole book, besides being illegal in most cases. 
Therefore, while this small program could accommodate a passing back and 
forth of materials, future programs should institute a rather formal 
library system . There are times when all trainees of a program would 
like to or need to look at some document or book at the same time, in 
spite of the fact that the program is individualized. In most instances, 
it is just not possible to have one copy of everything for everybody. 

Some system such as "overnight loans" would be recommended for future 
replications. We did not think six trainees would call for such a 
system, but there was a surprising demand for one. Informal communica- 
tions were not totally adequate in answering questions of what was 
available or who had what research report and when it would be back. 



The apprenticeship component 

In terms of exportability, the apprenticeship component has the same 
dependency on the particular agency and individual characteristics as 
does the staff portion of the instructional component. What takes place 
in an apprenticeship depends on the combination of individual trainee 
interest and/or expertise, and what is ongoing at the agency. However, 
again, any relatively large, well established agency engaged in educa- 
tional research and development would be able to make diverse offerings 
for apprenticeship activities. 

Therefore, while agency activities may provide some substantive boundaries 
to apprenticeship experiences, there are many common threads to the 
activities in all the potentially viable host institutions. What the 
apprentices experienced at AIR could be found at any number of other diverse 
research houses or agencies. The major modification would be found in 
the redirection of general apprenticeship activities toward specific projects 
within the agency. Thus, the apprenticeship component is highly exportable, 
although the partial definition of specific activities depends on the 
nature of the efforts within each research agency at the time of the 
training program. 



The job placement component 



The fourth component, job placement, is highly exportable and general izable 
as a concept and an activity. There were four major directions to job 
placement efforts. The first of these drew upon AIR itself as a potential 
employer, and AIR staff members' contacts with individuals in other agencies 
and organizations. While this phase had a direct bearing on AIR as the 
host institution, it is not necessarily unique to AIR. Any sizable research 
house is likely to offer the same potential opportunities, and its staff 
is likely to have personal contacts in a variety of other R&D organizations. 
In this sense, what was being taken advantage of was the R&D "fraternity," 
and this could be utilized at any site active in educational R&D. 

The second major direction of job placement activities was to keep abreast 
of the recipients of new contracts. This was done through an AIR in- 
house newsletter. Behavior Today , Review of Educational Research , and 
newspapers. The last three are publicly available documents which could 
be used at any site. The point here would be to contact these recipients 
of new contracts on the possibility that they would need additional staff 
members. 

The third job seeking effort was to encourage trainees to review the 
possibilities of their own contacts, such as relatives, neighbors, and 
so forth, who might be involved in education, educational R&D, or some 
related field. 

The fourth effort was the most traditional. Standard sources which posted 
job openings were examined regularly. These included the APA Employment 
Bulletin , the Educational Researcher , and civil service listings. In 
addition, resumes were sent "blind" to logical employers such as the 
Educational Laboratories. 

Of these four directions, the one which seemed to have the most substance , 
was personal contacts through AIR^ The educational R&D "fraternity" was 
more receptive and seemed to offer more solid leads than any of the other 
types of efforts. This, however, is not unique to this program or employ- 
ment seeking in general. It is widely acknowledged that in any type of 
job seeking activity, a personal reference or professional contact is 
usually more fruitful than just "knocking on the door." 

Surprisingly barren sources were the public listings, such as those of 
the APA Employment Bulletin . There were a few academic listings for 
university teaching jobs, and the like, almost none in educational R&D, 
and the offerings in general were very sparse. Again, this might have 
been an indicator of the relatively thin economic situation at the time. 

In summary, none of these efforts was unique to AIR, its location or its 
staff. While the personal contacts were certainly defined by the people 
who happened to be known, this procedure is not unique. In any setting 
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associations could and should be drawn on, for they appear to be viable 
sources. All of the other efforts are obviously public and could be 
utilized by any resourceful job seeker anywhere in the country. 



Comment 

A manual has been developed by the AIR staff to assist other institutions 
to replicate or modify the program of this study. (Millstein, E. , Melnotte, 
J., and Dunn, J., Procedures for Conducting an Apprenticeship Work-Study 
Program , Palo Alto! American Institutes for Research, 1972. Prepared 
under Office of Education Grant No. 0EG-0-71-4712.) 

It offers descriptions arid guidelines for the conduct of the four components 
the model . In addition, specific aids are included, such as instructional 
objectives, instructional resources and materials, samples of newspaper 
advertisements, press releases, letters of rejection and acceptance, an 
information booklet, and so forth. 



D. What is the trainee reaction to the program experience? 

Findings 



In general, the trainees were very pleased with the program and the oppor- 
tunity. As one trainee said, "The thirteen weeks was a most welcome 
opportunity to think deeply about education." All of the trainees felt 
they knew more about educational R&D than they did before the start of~ the 
program . Yhe pre- and post-test results, and the detailed self-report on 
specific objectives attainment verify this growth in knowledge about 
education and growth in performance levels on tasks associated with educa- 
tional R&D. 

However, interestingly enough, most seemed to agree that they could per- 
form the general tasks required for educational R&D before the start of 
the program, in spite of these indicators to the contrary. They per- 
ceived that the basic issue was problem solving, that they had been problem 
solvers in their past jobs, and that all that was now required was 
familiarity with a new field. What the program gave to them was not so 
much new skills as a different orientation. This view was strongly felt 
by all the trainees and is the most basic statement to be made of trainee 
reaction to the program experienced This was happily greeted by the 
project staff. The training program was not intended to teach new skills, 
but rather to transfer old skills to a new setting. The trainee reaction 
to the program, then, was very much in accord with the original philoso- 
phy of the program. 

There were a number of specific points to which the trainees had an oppor- 
tunity to react on the final trainee survey. These were: 

(1) Would you have desired more or less structure in terms 
of readings, formal seminars, or other types of 
directed study? 

(2) Would you have desired more mandatory activities, 
deadlines, etc.? 

(3) Do you feel you could or should have started on 
apprenticeships the first few days of the program? 

Examples of reactions to these issues are quoted below. 

*(1) Would you have desired more or less structure in terms of readings, 
formal seminars, or other types of directed study? 

"Just right as it was." 

"Preference for much more structure regarding 
readings and discussions, but in terms of 
independent study, . . . no." 
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"More structure for a shorter time." 



*(2) Would you have desired more mandatory activities, deadlines, etc.? 
"No." 

"I do not feel that there is a necessity for more 
mandatory activities." 

"No, except seminars on significant educational 
research topics." 

"The degree of structure was satisfactory." 

"Yes." 

*(3) Do you feel you could or should have started on apprenticeships 
the first few days of the program? 

"I could have done so, but felt it was desirable 
to delay apprenticeship work, in part the way 
it was done." 

"I do not feel that apprenticeship should be 
started in the first few days, but that 
emphasis should be on a more highly structur- 
ed orientation." 

"No." 

"That would be a very feasible approach for 
some people." 



Discussion 



Degree of structure 

As is obvious, individual reactions to specifics of the program differed 
with individuals. One trainee seemed fairly uncomfortable with indepen- 
dent study which was individualized, and would have desired a much more 
school-like setting. Others seemed fairly secure with this kind of approach. 
Still another trainee was comfortable with the approach, but probably would 
have been more industrious in a more structured setting. It is difficult, 
if not impossible, to predict a trainee's demand for structure from any 
application information, including the interview. 

All trainees were chosen because they appeared to be able to function com- 



fortably in a varied, but individualized setting, which required personal 
initiative, motivation, and judgment. An apprenticeship of this type in 
any work setting, is not equipped to do a great deal of "hand-holding." 
While it is highly desirable that trainees be willing to function in a 
highly independent manner, there seems to be no way to predict this in 
the selection process as employed, short of their personal expression, and 
this expression may not always be valid. 



Reading and study load 

Many of the trainees found the reading load to be quite heavy. As one 
pointed out, "It takes a great deal of personal discipline." However, 
this was not considered a great stumbling block, but rather a matter 
of adjustment. The majority of trainees either did not read a great 
deal in their past jobs, or had been unemployed for some time, where 
reading was done occasionally for leisure. The heavy reading and study 
load appeared to cause some discomfort. Again, this cannot be predicted 
from initial information. Almost anyone involved in educational research 
and development needs to be very comfortable with a lot of reading, writing, 
and, in general, verbal activities. It is difficult to screen out those 
who might be uncomfortable, but perhaps sel f-screening could be enhanced 
if the desirability of these skills was stressed in pre-application in- 
formation. 



Future employment 

Another major reaction to the program experience dealt with job-seeking 
and job location activities. This was discussed in a previous section, 
but it is appropriate here es well. It is probably safe to say that the 
trainees were disappointed that there were not a tremendous number of 
jobs available from which to select, or that AIR did not actually get all 
of them jobs. This could probably be classified as the major disappoint- 
ment of the program. It was stressed very early on that jobs would not be 
promised, were not part of the program, but that assistance would be offered. 
This was done. However, it is perceived that more optimism for jobs and 
more dependence on AIR was generated through program membership than was 
justified. 

The program staff felt that the possibility that jobs for trainees might 
not be obtained was well reinforced. However, perhaps it should be 
stressed even more in future programs, particularly when certain individuals 
are not being trained for certain pre-identified job slots. These were 
unemployed persons looking for new opportunities. That a new opportunity 
will actually be realized cannot be guaranteed in a program of this type. 
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E. What should be tfw content and p rooedures_ for follow-u^? 



The purpose of follow-up is obviously to find out what became of trainees 
once they had left the program and had gone their separate ways. Of 
special interest, of course, is whether or not they located .iobs in educa- 
tional R&D, and if not, whether they were still pursuing jobs in education- 
al R&D. Unemployment here becomes a rather special factor. It would not 
be surprising if trainees pursued all jobs, whether they were in education 
or not. It would be quite excusable if they returned to their old fields, 
or just sought any kind of job, for the sake of temporary employment. 
However, persistance in seeking educational R&D jobs would be one indica- 
tor of genuine interest in education, and success in locating positions 
would be an indicator of the degree to which the program creates a 
product with credibility and a "product" who functions comfortably in the 
new situation. 

The follow-up inquiry was conducted seven to eight weeks after termination 
of the formal program. There were six key questions of interest: 

(a) Have you sought jobs in educational research and 
development? 

(b) Who or what types of agencies have you contacted? 

(c) What have been your successes? 

(d) If you were turned down for a job or job interview, 
were you ever told why? 

(e) If you obtained employment in educational R&D, what 
have been your senses of satisfaction, confidence, 
and interest? 

(f) To what extent did membership in the program 
make a difference? 

Because of the small number of trainees, it was feasible to conduct this 
inquiry by telephone and record the major responses during the interview. 

If a larger pool is involved, it would be entirely reasonable to administer 
the questionnaire by mail. For those who would not respond by mail, a 
telephone inquiry would then be appropriate and feasible. 

This follow-up resulted in a variety of responses, even with a trainee group 
numbering only six. As indicated in a previous section, two trainees were 
permanently employed in educational R&D, one was rehired at his previous 
place of employment, and the other two trainees were still out of work. 

At the time the latter two were contacted, one had elected to not heavily 
engage in job-seeking activity at the present time due to illness in the 
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family. This particular trainee was from Seattle where unemployment 
benefits have recently been extended three months due to additional 
Federal support. This trainee felt that these extended benefits allowed 
for attention to mure immediate non-work related problems, but felt that 
once they were resolved, job-seeking activity in educational R&D would 
resume. There had been no response as yet to applications for employ- 
ment submitted during and immediately after the program. 

The other still unemployed trainee felt admittedly disillusioned that 
the program had not done more for him. It was felt that the job market 
must be particularly tight since it was difficult to even get an inter- 
view for any kind of job, and association with the program did not seem to 
enhance this for job inquiries even in firms whose activities were education 
related. This trainee expressed feelings of being caught in the middle. 

If jobs were sought in engineering, it seemed that interest in education, 
exhibited by being in the program, made eligibility and interest in engineer- 
ing jobs suspect. On the other hand, a strong engineering background made 
interest in education seem suspect* in spite of having been in the pro- 
gram. There appeared to be quite a strong feeling that program partici- 
pation was as much of a hindrance as a help, and that furthermore, not 
enough follow-up effort was made on the part of the program staff to 
set up post-program interviews. 

This disappointed view did not seem to be shared by any of the other 
trainees. As mentioned, the other trainee who was still out of work was 
not seeking work. The trainee who had been temporarily employed in educa- 
tional R&D was not now working, but also stated that there were not strong 
efforts directed toward job-seeking. Considerable effort, however, had 
been expended during the month of January. About fifteen organizations 
with educational R&D activities had been sent resumes. Most of these con- 
tacts were suggested by the survey list of educational R&D employers con- 
structed by Far West Laboratories. As of mid-February, the trainee had 
received about six negative replies and one positive offer in Florida. 

Efforts at job-seeking had slowed somewhat due to a decision to switch 
graduate programs from an M.S. in Psychology to an M.S. in Educational 
Psychology. The necessary courses in the transfer would begin in the 
early summer, so that job decisions might be delayed until after that 
program was complete. 

Three of the trainees are currently employed. The one who was rehired at 
Boeing is still in his old position and feels relatively secure. The 
statement was made during the interview that the program at AIR was highly 
enjoyable and that work in education was most attractive. It was deemed 
regrettable to have to leave the program, but it was felt that the oppor- 
tunity for certain employment had to be taken when it was offered. 

The other two employed trainees are now in work related to education. 

Both are extremely satisfied with their jobs. One in particular feels 
that the work is varied, that he has considerable mobility in his immedi- 



ate environment, and very much enjoys the people with whom he is working. 

He feels that the work being done and the environment he is in are not 
totally strange, and that the program contributed to these feelings of 
confidence and familiarity. In addition, the program was given credit 
as contributing a considerable degree to his credibility in obtaining 
the job contact and in getting his job. While he could perform his job 
to a certain degree without the program, the actual experience of the 
program offered him considerably more "savvy" about education than he 
might otherwise have had. It was not a matter of learning the job from 
the ground up with or without the program, but the program enhanced a 
quick and easy transfer of his skills from one work environment to an- 
other. 

All of the trainees had received rejections to their applications in 
one form or another. In every case, those rejections consisted of 
obvious variations of form letters. Comments such as, "We have reviewed 
your application and were impressed with your qualifications, but ..." 
were quite common. In no case could it be determined exactly why a 
trainee was not interviewed or not hired. However, a number of letters 
did state that the organizations simply were not hiring at this time. 

Given the current economic health of education, this could well be the 
case. 

These follow-up procedures worked very nicely with this group of trainees. 
Each telephone interview averaged ten to fifteen minutes and considerable 
information was gathered. The questions seemed appropriate, to the point, 
and answerable. As to the timing of follow-up, anywhere from four to 
six weeks after termination of the program would seem the most fruitful. 
Time periods shorter than this do not really allow for placement activities 
to offer stable results; and in a longer time frame, trainees are likely 
to lose contact and identification with the program. 

These procedures, or variations of the same procedures, are adaptable 
and highly recommended for future programs. 



O 
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F . How responsive is the manpowe r _££ol? 

Background discussion 

Two basic strategies were employed to make the program's existence known 
to the potential manpower pool. These included the placement of an ad and 
a press release in key newspapers and a public posting of the program in 
Seattle and Bay Area Employment Service Offices. In both these areas, there 
are a large number of major and minor newspapers, trade papers, ethnic 
papers, and so forth. Since there were not the funds and, perhaps, not 
the necessity, to place the ad and press release in all papers, decisions 
needed to be made about which of the myriad of sources would be most rele- 
vant. This was particularly true of the ad, where money was a factor. 

These decisions were based on three considerations: 1) audited circulation 

of the newspaper; 2) proximity to the likely pocket of unemployed aerospace 
personnel; and 3) local notoriety that the paper's classified section was 
a popular reference for job-seekers. Based on these combined factors, the 
ad was placed in The Seattle Times (Thursday and Sunday editions). The 
San Jose Mercury-News (Sunday edition) and the joint classified section 
of The San Francisco Examiner ar. i The San Francisco Chronicle (Thursday 
edition) . 

A press release was also issued to these and other regional newspapers. 

These included The Seattle Post-Intelligencer, The Seattle Times, The 
Tacoma News-Tribune, The San Franciso Chroncle, The San Francisco Examiner, 
The Oakland Tribune, The Palo Alto Times, and The San Jose Mercury-News. 

Tiie press release, however, is not a reliable news dissemination vehicle, 
because the perscn who issues the release has no power or control over 
whether or not the paper will print it. In addition, even if the release 
is printed, it is likely to have been edited, which may or may not change 
the flavor of the message. Therefore, primary reliance for getting responses 
was placed on the ad, and secondary reliance on the Employment Service 
posting. 

Persons who saw the ad or posting were then to make telephone contact with 
either our Seattle representatives at the University of Washington or with 
AIR directly. This, of course, depended on where the potential applicant 
lived. The purpose of the call was for each potential applicant to get 
additional information and to then request an application form if the 
program still seemed attractive to him. Without some sort of defined time 
period for responding, calls could obviously come in for quite a number 
of days or even weeks. Therefore, the formal publicized response period 
was established at three days for each site. 



Initial response results 

Before and during the establishment of this process, there was absolutely 
no way to predict what the response would be. However, it seemed a logical 
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process, and our guesses were that the response would be substantial, but 
we had no way of knowing for sure. 



The results were as follows: 








Number of initial calls in 


Seattle 


Bay Area 


Total 


response to the ads end 
Employment Service postings 


160 


114 


282 



Given the minimal recruiting efforts of a simple newspaper ad and an in- 
formational posting in unemployment offices, this response is considered 
substantial. This is especially true in light of the fact that the ad 
contained sufficient information to pre-screen many potential applicants 
on a self-select basis. In addition, had any more calls been received 
during that time period, they simply could not have been handled. Given 
a full work day, calls were received at an average rate of one every seven 
minutes in Seattle and one every eleven minutes in Palo Alto. By the time 
information was exchanged, addresses taken, and so forth, the phone was 
ringing again. In fact, it would have been difficult to handle many more 
calls in that time period without expanded reception facilities. The two 
telephone lines in Seattle were in almost constant use, and the switch- 
board at AIR was extremely busy. 

It would be logical that since the program was based in the Bay Area, and 
since the Bay Area has over three times the population of the Seattle area, 
there would have been more Bay Area inquiries. As indicated, this was 
not the case. One explanation might be that Seattle had, and still does 
have, a more severe unemployment situation. The Bay Area has a consider- 
ably more diverse employment market than Seattle, which is heavily involved 
in aerospace as a total community. This difference was reflected in their 
respective unemployment rates: as of June 1, 1971, the Seattle unemploy- 

ment rate was 13.9%, while the rate for Santa Clara County (center of 
Bay Area aerospace activities ) was 6.3%. This difference is no doubt re- 
flected in the relative number of calls received in the two areas. 




Interested inquiries 

The next point to be examined was whether or not callers were still in- 
terested in receiving an. appl ication after hearing more information about 
the program, particularly with reference to money. While the ad gave 
considerable information, it intentionally omitted any reference to the 
stipend. The reason for this was to determine whether the very modest 
amount discouraged further interest, which would be exhibited by not re- 
questing an application. 
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The results of this inquiry were: 



The number of inquirers who Seattl e Bay Area Tot al 
requested or were sent the 

application and fact book- 168 102 270 

let after the brief tele- 
phone contact 



Percent of initial number 

of inquiries 100% 90% 96% 

Strangely enough, the issue of money was very much in the background. Out 
of 282 calls, only one person decided not to pursue application because 
of the small stipend. In addition, the Bay Area figure is somewhat mis- 
leading. While 102 applications were requested from a total of 114 original 
inquiries, 10 of that difference of 12 were due to non-eligibility. For 
the most part, they were people who knew they were not eligible from in- 
formation in the ad, but were trying nonetheless. These included teachers, 
persons without degrees, and so forth. Therefore, out of 282 initial 
inquiries, only 2 elected not to pursue further application procedures 
given the additional information. Of these two, one opted out because of 
the modest stipend, as indicated, and the other would not say why there 
was disinterest. Therefore, the real response percentage at this stage 
was 99%, rather than 96%, a substantial figure, indeed. 



Number of applicants 

At this point, the initial call rate was known, and the number of persons 
still interested enough to obtain an application was also known. The 
next stage was to examine the number of persons who actually followed through 
on submitting the application. This, perhaps, was considered the real test 
of interest in the potential manpower pool, for along with the application 
was sent a fact booklet which discussed all the salient points of the 
program in question and answer format. (The contents of that booklet are 
offered in Appendix I of the companion final report on the program itself.) 

The results were: 



Number who actually 


Seattle 


Bay Area 


Total 


returned the application 


102 


73 


175 


Percent of those who 


had received applications 


61% 


72% 


65% 



It is difficult to state what percentages would represent "overwhelming" 
response, "moderate" response, etc., since there is no precedent for 
comparison. However, the facts that no more initial calls could have been 
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comfortably handled, that almost 10 out of 10 persons who initially in- 
quired chose to receive an application, and that approximately 6.5 of those 
10 actually followed through with submitting the application all point to 
what might be labeled extremely strong response. When only 6 of 175 
applicants could be selected, there was certainly no shortage in the 
numbers of persons from whom selection was to be made. 



Women applicants 

Of special interest during this recruitment process was the number of 
women who applied. These results were as follows: 



Numbers of male and 


Male 


Seattle 

96" 


Bay Area 
70 


Total 

166 


female applicants 


Femal e 


6 


3 


9 


Percent of female 


applicants 




5.9% 


4.1% 


5.1% 



Strong affirmative action was taken to secure female applicants. However, 
applications from females accounted for only 5.9% of the Seattle pool, 

4.1% of the Bay Area pool, and 5.1% of the total. 

Data on the total female market, unemployed due to aerospace cutbacks in 
the two regions, were not readily available. However, in an attempt to 
make some judgment about these proportions of female respondents, the 
Santa Clara County Unemployment Office was contacted, and they agreed to 
summarize the data from their files. As of August 23, 1971, they had regis- 
tered 662 engineers, scientists and technicians who had been laid off from 
companies involved in aerospace. Of these, only 5, or less than 1%, were 
females. If it is inferred that those figures are representative of the 
relative size of the male and female markets available in Santa Clara 
County, and perhaps, other counties, then the 4% or 5% female application 
response exceeds expectations. 

As a side corrment, it might be mentioned that the nine females who did 
respond were all relatively strong candidates, and as a group, more 
homogeneously strong than were the men. If a statistical analogy might 
be made, the females had a generally higher "mean" and a smaller "standard 
deviation" than did the males as a group. However, there are the obvious 
weaknesses of comparing groups of 166 and 9 respectively. These are only 
intended to be preliminary observations. 



In summary, then, the manpower pool was very responsive, and the vehicle 
of a newspaper advertisement to publicize the program was most effective. 
It should be kept in mind, however, that the want ad section of the 
newspaper was particularly appropriate for a target audience of unem- 
ployed aerospace personnel. A different audience would require different 
techniques and different sources. As an obvious generalization, the 
vehicles for information dissemination need to be tailor-rriade for the 
particular group of trainees desired. Key questions would include 
"Where are they most likely to read or hear about our program," and 
"What are the most likely sources for contact." Given the audience 
in this particular case, newspaper want ads and Employment Service 
offices were highly logical sources and highly effective sources. 



G. What procedures should be_ foltowed for the seleation o£_ trainees ? 

In summary, the selection steps were as follows: a) the screening for 
eligibility and rating on a five point scale the 102 applications from 
the Seattle area and the 73 applications from the Bay Area; b) selecting 
the top-rated applications which numbered about two dozen from both areas; 
c) the contacting of those top-rated applicants to come to either the 
University of Washington in Seattle or AIR in Palo Alto for additional 
screening; d) administering an educational areas test and obtaining a writ- 
ing sample from the top-rated applicants in each area; e) scoring the 
test and the writing sample; f) selecting the top dozen from each area for 
personal interviews; g) conducting interviews; h) selecting the top six 
from each area on the basis of cumulative information; i) review and 
ranking of the top six from each area by AIR program directors; and 
j) final identification of the three from each area selected for the 
program, plus three alternates from each area. 

This general paradigm of application, testing, and interviewing represents 
three screening stages, each with finer decision-making delineation. 

There were four major decision steps, i.e., 1) toward what population will 
the applications be aimed? 2) of those applications, who will be selected 
for further testing? 3) based on applications and test results, who will 
be selected for interviewing? and 4) who will be selected for the program? 
Only the first of these was pre-defined by the very nature and intent of the 
program. The decision had already been made to conduct the program for un- 
employed aerospace personnel. However, the other three decision stages 
had to be and were conducted in their various appropriate time frames as 
the results of the selection process required more and more refinement. 

This general paradigm was, perhaps, quite traditional, but it was also 
efficient. A great deal of data were processed, and decisions on that 
data were made in a fairly short period of time. The judges consisted of 
program staff in the early stages of the process, and program staff plus 
AIR program directors at the later stages of the process. There were no 
major rating discrepancies between and among judges at any of the stages. 
This was particularly true of the initial steps of the application review 
where independent ratings were made on each of the 175 applications. 

There was a somewhat lesser degree of agreement only at the final stage of 
selecting the trainees from those interviewed, and even these were not 
gross disagreements. The staff would have been comfortable with most of 
those interviewed: the issue was how to select the best, from an already 

agreed upon, good pool. 

It was found that the rating and test data provided fairly natural breaking 
points. It may have been serendiptous , but these points nicely accommo- 
dated the original notion of testing the top two dozen from each area, 
interviewing the top dozen, and selecting three trainees, with three al- 
ternates from each area. In summary, the general paradigm was considered 
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appropriate and successful, although not necessarily unique. The selection 
process was efficient, smooth, and fairly natural. 



Application 

As is true in any endeavor of this type, certain points of the selection 
process provided more information than others. The application was a 
major source of information and the staff would be more than willing to 
use it again. The major weaknesses were in the area of credibility on such 
issues as genuine interest in educational P.&D, acceptable salary, and will- 
ingness to relocate. But these are issues in any case, no matter what 
inquiry techniques are used. 



Writing sample 

The writing sample was a more than reasonable screening device, given its 
pervasiveness in educational R&D. However, in this situation the degree to 
which pressure affected quality is indete.minant. One factor which may 
have played a role is that the question was pre-determined, applied to all 
candidates, and was fairly specific. This was deemed desirable for purposes 
of standardization and consistency. However, in order that content not 
confound style and organization of thought, which are the real issues, it 
is recommended that the topic on which a condi date writes be somewhat less 
structured. 



Test 



The ETS test did offer a good distribution of test scores, and it did play 
a role, along with the writing sample, in who would be interviewed. However, 
there were yet to be answered questions about the role of tests specifically 
aimed at educational areas in trainee selection, and whether this particu- 
lar test was most appropriate. It is asserteo that the test was general 
enough in nature that a reasonably involved, reasonably intelligent, rea- 
sonably well read person could perform adequately on the test, whether or 
not his past experience involved educational issues and information. It 
is true that trainees did exhibit stronger performance on the post-program 
administration of the same test. However, this is quite a different issue 
as to whether or not it is desirable in the selection phase of the program. 

However, some objective screening device is desirable, if only to give 
program staff some common, well known, acknowledged reference points. At 
issue is the extent to which that test needs to be specifically associated 
with education or whether some other type of measure would suffice. This 
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issue is further discussed Section J, but in summary, it is now felt that 
any general instrument which assesses the ability to function effectively 
in a verbally oriented learning environment would suffice just as well, and 
perhaps even better. The generalized portion of the Graduate Record Examina- 
tion, College Entrance Examinations, or the like, are good examples. The 
fact that a trainee have a strong pre-program knowledge of education does 
not seem to be crucial. It would seem far more important that the trainee 
5e adaptable and more than reasonably bright, no matter what the area o T 
expertise. 



Interview 



The interview is the most interesting and enlightening component of the 
selection process. If a program had the time and money, it might be fruit- 
ful to interview all potential trainees, save the obviously unqualified. 

One problem with interviews is that they can be misleading: an "interview- 
wise" candidate can offer a variety of socially acceptable responses which 
may or may not be a true representation of actual position. The interview 
process in this program's selection was usually conducted by one person, 
with one interviewee, and was fairly structured. There are two recommenda- 
tions which might now be made. Firstly, the interview should be conducted 
by more than one person , which might roughly be labeled a panel. The 
early stages of the interview should be highly structured and conducted 
around very specific questions. The latter stages of the interview should 
be unstructured with the questions or topics emanating from responses in 
the structured portions of the interview. This process was very much like 
that utilized in this program. However, it was felt that more than one 
interviewer would reduce errors of judgment and could made additional, solid 
inputs for information gathering. In general, there was more confidence 
in a consensus opinion than a single opinion. 

A second recommendation, although perhaps not as realistic, is that there 
be some sort of mass interviewing of all potential trainees with all senior 
program staff. This process has the appearance of an unstructured discussion 
group, but is, in reality, quite structured. The Rhodes Scholar Selection 
Committee has applied this technique and found it to be quite successful. 

It has the advantages of efficiency in time, the provision of more life- 
like and work-like settings, and minimizes the impact or import of individual 
errors of judgment. The disadvantages are that it requires a great deal of 
time and sophistication to set up such a studied, calculated interview, with 
none of the apparent characteristics of a studied, calculated interview. 

In addition, it can be expensive, for such techniques usually require a 
degree of social entertaining. 

Therefore, while the latter approach is attractive, it is perhaps not feasi- 
ble and not worth the effort and money for short-term, relatively small 
scale programs. Therefore, the most realistic recommendation at this time 



is that of the panel interview as described. 



In summary, the application process was effective as designed, the writing 
sample should be more individually selected, the testing phase could be in 
more generalized areas, and the interview stage should involve more than 
one staff member. What was done was by no means deemed to be ineffective, 
or to have lead to inappropriate decisions; but the changes in the selection 
process, as recommended, are considered to potentially contribute to easier 
decisions with a greater degree of confidence. 



H • What are the implications to the host institution in conducting an 
appre nticeship travnvng program? 

Trainee presence 

Senior staff who had had contact with the trainees were periodically inter- 
viewed to assess the degree to which performance of, and interaction with, 
the trainees had been satisfactory. An important concern before the program 
began was the extent to which the presence of six relatively naive strangers, 
working throughout AIR, would interfere with established routines, require 
an inordinate amount of supervisory and explanatory time, and generally be 
disruptive to an ongoing, efficient work pattern. In general, these fears 
did not materialize. In no way did the staff feel that the trainees were 
an intrusion . Comments included, "they seem to be adult and independent 
workers," "having people around with different backgrounds is refreshing," 
and "they reinforce my tutorial inclinations." 

On the other hand, while the trainees were by no means an intrusion, neither 
were they a strong professional advantage. It was originally felt before 
the program started that having six technically and professionally compe- 
tent additions to the working environment would provide the impetus for 
new, fresh ideas, new ways of approaching old problems, and perhaps new 
techniques and processes. It was felt that we might learn something new 
about processes and problem solving in the physical sciences which could 
be advantageously adapted to the behavioral sciences. However, this did 
not seem to materialize. From an AIR point of view, one, maybe two, trainees 
brought new skills and new ideas to educational R&D. The other trainees 
were akin to junior staff members already in educational R&D. It is, of 
course, entirely possible that the specific apprenticeship activities and 
the relatively short duration of the program did not allow the manifesta- 
tion or maturation of potentially valuable inputs. But, given the fairly 
short time span of the program, it can be observed that having been in 
the physical sciences, having performed different tasks in a different 
environment, did not seem to make a startling difference in professional 
contributions to education. Many of the trainees did, in fact, feel that 
"work was very similar— it was just done in a different area." As a 
general statement, the deoreee of creative innovation provided by profession- 
als. from a different field did not meet expectations. While new ideas and 
approaches were certainly offered, they seemed to be more functions of 
the individual person rather than their particular work experience. These 
people would be likely to bring creativity and industriousness to any 
setting. The mere fact of having the opportunity to reapply old skills to 
a new area does not necessarily result in new contributions. 

Therefore, the host institution which expects a great influx of potent man- 
power for a limited period of time in the conduct of such a program, simply 
cannot count on it. As in any work situation, in any field, it is still 
a matter of personal contribution. Perhaps it could be stated that "work 
is work," and individual contributions are made in individual settings. 
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Therefore, while the trainees were by no means an intrusion, neither were 
they a great professional advantage. 



Staff commitments 



A minor problem area is the degree to which staff in the host institution, 
who are not directly involved with the training program, can be drawn upon 
to give seminars, lectures, administrative time, and so forth. For example, 
the AIR staff provided a rich resource for professional guidance and expertise, 
but time was at a premium. A large number of staff did make contributions 
to the program, but these contributions were in addition to an already heavy 
work load. The issue is not money. That is, it would make no difference 
if outside-program staff within the institution were paid as "consultants" 
to conduct a seminar, for example. A budget provision for these areas 
would not resolve the problem. The problem is one of time. In most of the 
host institutions, staff would be very committed to conducting their work 
on other contracts or projects. To conduct a one-hour seminar probably 
takes at least four to eight hours preparation time. When there are many 
other commitments and deadlines to meet, this much time can be quite 
demanding. 

The situation is certainly not impossible, however. In-house expertise 
is highly desirable and even necessary to programs of this type. The 
program at AIR did not meet with many difficulties: staff were willing 

and happy to participate. But there are a number of prerequisites which 
pave the way for cooperation. First, outside staff need to know the 
purpose of the program and understand the role of the host institution in 
conducting the program. Secondly, they need to know why they, as individuals, 
are needed to discuss their particular area of expertise. And thirdly, they 
need a reasonable amount of lead-time. These seem to be common sense fiats, 
but they are very basic. In all but the extreme cases, these procedures 
should enhance willing cooperation and enthusiastic participation in the 
program. In the case of AIR, staff who were brought in to participate in 
the program felt tha* it was a happy and worthwhile venture. 



Administrative considerations 



General administrative procedures proved to be of little trouble for the 
host institution . These administrative procedures included such tasks as 
registering trainee Social Security numbers, arranging for payments of 
stipends, finding desks and office space, and the like. While it took a 
bit of time for the immediate program staff, it was certainly relatively 
minor. AIR has a well developed system of office services and accounting 
which contributed to the ease with which these tasks were accomplished. 
This would no doubt be true of any established organization. 
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I • What are effective £rooedures for the moni toring of trainee progress? 

Need for documentation 



First and foremost on this issue, not enough can be said for documentation. 
Trainees need to record and formally submit their activities and reactions, 
and staff need to maintain records of assignments, known trainee activities, 
personal reactions, and reactions of staff members who are not officially 
participating in the program, but who have had contact with the program. 

It is a very sobering thought that some programs might be conducted without 
such documentation. Not all of what transpired could be mentally stored and 
recollected by one or even half a dozen persons, even on a weekly basis. 

In order to be fully aware of the dynamics and activities of any program 
of this type, there needs to be an almost daily accounting of events and 
observations. This was done by both the director of the training program 
and the director of evaluation. "Keeping track" is vital, and cannot 
be successfully accomplished without persistant attention and formal 
recording. 



Approaches to monitoring 

Three major recording approaches were employed, and they proved to be high- 
ly successful. The responsibility for two of these approaches lay with the 
immediate project staff. 

First, the director of the program and the director of evaluation of the 
program maintained personal daily "diaries." These included notations of 
trainee assignments and accomplishments, staff meetings, and even personal 
affect to events of the day. 

Secondly, these two staff members recorded all interviews with outside- 
program AIR staff who were interacting with the trainees, and with the 
trainees themselves. These two processes provided an excellent, comprehen- 
sive overview of what had transpired over the thirteen-week period. With- 
out such documentation, it would have been impossible to maintain a realistic 
picture of perceived progress, with all its nuances of change and growth. 

The particular technique involved in this area was fairly structured. The 
director of the program made a point of interviewing each trainee twice 
weekly. The intent of these meetings was to determine progress to date, 
determine future objectives, suggest possible strategies to meet those 
objectives, and register various areas of satisfaction or difficulty. 

This sort of information gathering required a great deal of time and per- 
sonal attention from the project director. Since this project involved 
only six persons, this extensive degree of personal attention was feasible. 
However, programs on a much larger scale could not employ quite the same 
approach. 
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There would have to be either 1) more staff members involved in such 
personal, individualized contact, 2) one person whose only task was to 
monitor progress and discuss activities with the trainees, or 3) some 
system which employed an "ombudsman," to whom trainees could turn for the 
resolution of particular problems or the reception of new tasks. Of 
these three, the second is probably the most viable. Monitoring progress 
of a sizable number of trainees could be a full-time task. In addition, 
there would be the advantages of having a central resource person who was 
aware of the total situation, and the provision of something more than 
just informal, occasional contact when there were problems. Additional 
staff members involved in this task could make for disparate communications 
and an ombudsman would have an unpredictable and undefined work load. 

In summary, personal monitoring is of high necessity. If a program is 
small, the project director could probably handle it satisfactorily. If 
a program is large, there would need to be a special position of some sort. 

The third recording approach was the responsibility of the trainees and also 
proved to be highly successful. During the first days of the training 
program, it was announced that trainees would be required to keep an in- 
dividual daily chronicle of events— what they did that day, with whom they 
talked, what they read, their pleasures and disappointments. These docu- 
ments, referred to as "logs," were turned in each Monday morning to the 
director of evaluation for review and summary. The logs provided an in- 
valuable and unique source of information on each trainee's activities 
and academic pursuits. It was an obvious monitoring device as to the 
meeting of assignments and the identification of individual efforts at 
following up expressed areas of interest. Since each trainee maintained 
the log on a daily basis and accumulated them, it provided each trainee 
with a personal view of progress. In addition, since the logs were also 
summarized and annotated by the program staff, it served as the basis 
for a cumulative file on each trainee. 

The format of these logs was an interesting point for decision. During 
the first half of the program, the logs were very "open-ended." Trainees 
were simply asked to record their activities and reactions in any form which 
was comfortable. Obviously, these logs were mostly of the essay type and 
varied considerably as to the information they provided. Some trainees 
seemed to have considerable difficulty expressing affect. Either they did 
not know in what areas there was interest in their reactions or they did 
not know how to express them. At the midpoint of the program, an alter- 
nate approach to "log keeping" was instituted. The new format minimized 
open-ended responses and maximized the specificity of desired information. 
However, the opportunity for "conments" was maintained and encouraged. 

The new format defined the areas of response through specific questions. 

They were: 

(1) Apprenticeship activities? (What? For whom? When?) 
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(2) AIR staff contacts and general purpose of meeting? 

(3) Readings? (Please indicate whether related to (a) modules 
or other assignments, (b) apprenticeship activites, (c) 
personal follow-up on a topic of interest.) 

(4) Job search activites? (sources identified, resumes up- 
dated, letters sent, interviews set up, outcomes, etc.) 

(5) Transfer skills and new skills which would be attractive 
to potential educational R&D employers? 

(6) Comments (e.g., suggestions for program, accessibility of 
materials, reactions to apprenticeship activities, etc.) 

It was felt that the old, "looser" format might not be amenable to gathering 
the desired information. Trainees generally seemed uncomfortable with it 
and relatively insecure about what to record. In the monitoring of the logs, 
it was felt that the information offered in the loose format was fairly 
substantial, although the trainees needed a good deal of encouragement 
to express personal reactions to events. 

It was deemed that the new, directed format would provide an even richer 
source of information, since the communication of desired information 
seemed so clear. However, it did not turn out that way. The new format 
resulted in amazingly restricted documentation. Those who had been quite 
verbal under the old format now gave rather cursory answers to the specific 
questions. Those who had been terse in the first place found the new format 
an opportunity to be even more terse. Therefore, it is highly recommended 
that the form of future personal documentation by the trainees follow the 
format originally employed in the program; that is, open-ended and relatively 
free flowing. Verbal guides were, of course, initially offered, but after 
that it seemed more fruitful to allow the trainee to offer expressions of 
events which seemed most personally important, plus some attention to the 
verbal guidelines offered by staff. It might be feared that under these 
conditions, program monitors would not get the information they needed, but 
this did not seem to be the case. 



In summary then, personal contact ir monitoring trainee progress seems a 
must. It is quite time consuming, but future programs should provide for 
that. In addition, careful attention to documentation is also a must, both 
from the trainee viewpoint and the staff viewpoint. To rely on memory and 
casual month-to-month judgment would be entirely insufficient. There are 
just too many factors which need recording to support such a casual approach. 
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orientation. They would no doubt be more concerned with developing effec- 
tive workers than knowledgeahle scholars. To date, there seem to be no 
multi -phasic criterion-referenced tests to measure the effectiveness of 
educational R&D workers. This may well be due to the fact that the criteria 
have not been fully explicated. Until that time then, the use of objective 
measures to assess the degree to which educational R&D workers are effective 
workers seems unfeasible. In the meantime there will need to be the tradi- 
tional reliance on opinion, productivity, the satisfaction of critical 
incidents and the like, all of which are likely to be situation-specific 
and involve personalities. The state of the art has simply not progressed 
to the stage where more objective alternatives are available. 



K. Who agglied^ What were their dha^aateristios ? 



The following tables summarize the information on selected characteristics 
for both the Seattle and Bay Area pools of applicants, plus a total for 
both pools combined. In some cases, the n's may not total the number of 
applications received, due to omitted information. 



A^e 



Range 


Seattle 


Bay Area 


Total 


25-29 


11 


3 


14 


30-34 


7 


9 


16 


35-39 


19 


13 


32 


40-44 


14 


13 


27 


45-49 


18 


14 


32 


50-54 


21 


7 


28 


55-59 


9 


9 


18 


60-64 


3 


4 


7 


? 




1 


1 


X 


43.5 


43.9 


43.6 



The average applicant might be described as "middle-aged." It was somewhat 
surprising that the group was as mature as it was. Originally, it was 
thought that younger, less experienced aerospace personnel would be more 
likely to receive the impact of cutbacks and thus would have disproportion- 
ate representation in the applicant group. As a general observation about 
the group as a whole, a surprising number of persons seem willing to trans- 
fer careers at a point which is past the middle of their working lives. This 
statement may, of course, be tempered by the urgency of their situations. 

It could well be that another target group, which did not totally consist 
of the unemployed and was not solely from aerospace, would have quite a 
different configuration. 



Sex 





Seattl e 


Bay Area 


Total 


Male 


96 


70 


166 


Female 


6 


3 


9 



The number of female applicants was considered to be quite small, since 
every effort was made to encourage this group. However, as discussed in 
Section F, these numbers may bear some relationship to male- female differences 
in the general population of unemployed aerospace personnel. There is some 
evidence that an extremely high percentage of this population is male. 
Therefore, while this number of females appears low, it may in fact be more 
than reasonable, given the pool. Obviously, this configuration would be 
quite different, too, if the target population had been, say, unemployed 
teachers in California. Therefore, while substantial female representation 
was highly sought, there simply may not have been that representation in 
the target pool, given the criteria. 



Years of School Past B.S. or B.A. 



Years 


Seattle 


Bay Area 


Total 


0 


62.7% 


38.9% 


52.9% 


Up to 1 yr. 


22.5% 


11.1% 


17.8% 


Up to 2 yrs. 


12.7% 


29.2% 


19.5% 


Up to 3 yrs. 


1.0% 


11.1% 


5.2% 


Up to 4+ yrs. 


1.0% 


9.7% 


4.6% 



O While almost half of the total group had done some graduate work, the Bay 

ERJC Area group accounted for a greater portion than did the Seattle group. When 

long-term graduate efforts are inspected ( i . e. , up to 3 years or up to 4+ 
vearsl. the Bav Area arouD had aDDroximatel v ten times the reoresentation of 



Graduate Degrees Earned 



Degree 


Seattle 


Bay Area 


Total 


M.A. /M.S. 


9.8% 


27.8% 


17.2% 


M.B.A. 


2.0% 


9.7% 


5.2% 


Ph.D. 


1.0% 


11.1% 


. 5.2% 


Totals: Any 








grad, degree 


12.7% 


48.6% 


27.6% 



Half again as many of the Seattle group as the Bay Area group had 
some graduate study. But about four times as many of the Bay Area group 
had actually completed the requirements for and received a graduate degree 
of some type. There were about three times as many M.A. or M.S. recipients, 

4 to 5 times as many M.B.A.'s, and over ten times as many Ph.D.'s who applied 
from the Bay Area. Again, the Bay Area group of applicants appeared to be 
more academically high-powered than did the Seattle group. Much of this 
no doubt has to do with the graduate degree demands in the respective labor 
markets. 



Last Salary (in Aerospace Industry) 



Range 


Seattle 


Bay Area 


Total 


$ 6,000 - $ 7,999 


4.0% 


2.8% 


3.5% 


$ 8,000 - $ 9,999 


10.0% 


12.7% 


11.1% 


$10,000 - $11,999 


16.0% 


12.7% 


14.6% 


$12,000 - $13,999 


20.0% 


31.0% 


24.6% 


$14,000 - $15,999 


24.0% 


16.9% 


21.1% 


$16,000 - $17,999 


12.0% 


14.1% 


12.9% 


$18,000+ 


14.0% 


9.9% 


12.3% 


X 


$14,386 


$14,544 


$14,451 



Acceptable Salary (in Educational Research and Development) 



Range 


Seattle 


Bay Area 


Total 


Open 


14.3% 


29.0% 


20.4% 


$ 6,000 - $ 7,999 


6.1% 


2.9% 


4.8% 


$ 8,000 - $ 9,999 


27.6% 


15.9% 


22.8% 


$10,000 - $11,999 


25.5% 


29.0% 


26.9% 


$12,000 - $13,999 


19.4% 


14.5% 


17.4% 


$14,000 - $15,999 


5.1% 


8.7% 


6.6% 


$16,000 - $17,999 


1.0% 


i 


.6% 


$18,000+ 


1.0% 


- 


.6% 
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Interestingly enough, the salaries represented in the two areas are relative- 
ly comparable. The fact that the Bay Area group was a better educated group 
did not seem to make much difference when it came to money. 

In general, the applicants represented a group which had enjoyed reasonably 
comfortable salaries. The average salary during aerospace employment was 
$14,451: about half the group had salaries exceeding $14,000 a year. 

However, as indicated, they were willing to earn considerably less 
as starting salaries in educational research and development: less than 

8% of the total group expressed a figure of $14,000 or above as an acceptable 
level . 

It is difficult to tell how much of this was genuine and how much was prompted 
by statements in the Fact Booklet originally forwarded with the applications. 



One section of the booklet dealt with reasonable salary expectation should 
an applicant enter the field of educational R&D. Those figures ranged 
from $8,000 to $12,000, with a somewhat higher range for special skill 
areas. Many applicants may, of course, have recorded what they deemed 
would be "acceptable" in the eyes of those reviewing the applications. 



Months of Unemployment 



Months 


Seattle 


Bay Area 


Total 


0- 4 


22.8% 


38.4% 


29.3% 


5- 9 


35.6% 


16.4% 


27.6% 


10-14 


10.9% 


23.3% 


16.1% 


15-19 


20.8% 


12.3% 


17.2% 


20-24 


7.9% 


5.5% 


6.9% 


25-29 


2.0% 


2.7% 


2.3% 


30+ 


- 


1.4% 


.6% 



Prior to seeing these figures, it might have been hypothesized that since 
the Seattle unemployment situation was more severe than that of the Bay 
Area, there would be a much higher proportion of long-term unemployment in 
the Seattle group. However, this difference was not quite as extreme as 
would be imagined. About 31% of the Seattle group versus 22% of the Bay 
Area group had been out of work for fifteen months or longer. In fact, 
a small proportion of the Bay Area group had been unemployed for an 
astoundingly long 2h years or more. No doubt these situations provide 
strong impetus for seeking work in other areas, perhaps in any area, and 
accepting salaries which are lower than those to which they had been 
accustomed. 



L. What were reasons for a^^Hoant rejection ? 

Appropriateness of initial criteria 

The original criteria on which applications were rated included evidence of 
interest in education, existence of specific personal career objectives 
within the field of education, reasonable salary expectations, evidence of 
success in college, and evidence of success in employment. Of these, the 
one which was least easily applied was "existence of specific personal 
career objectives within the field of education." Candidates were judged 
as to what appeared to be strong commitment and interest in education and 
the extent to which they could offer general topical objectives, but to 
assess specific objectives was not reasonable. Most applicants were not 
in a knowledge position to be very specific and, perhaps, chose not to express 
much specificity for fear of limiting their opportunities in the training 
program. 



Additional criteria 



There were additional criteria which were or became part of the system. One 
of these was owning a B.S. or B.A. degree. A number of persons applied 
without the degree, in spite of this specific citation in the newspaper 
advertisement and Employment Service postings, on the argument that they had 
an equivalent amount of training. The decision was to hold to the original 
requirements, "as advertised." 

Another unexpected criterion became that of "appropriate" qualifications, in 
the sense that some applicants were deemed almost over-qualified and did 
not "need" the program. As an example, one applicant had been heavily in- 
volved in education in his last job, had been vice president of a major 
private research house, held a Ph.D., and his last salary was $35,000 per 
year. To bring him to AIR, at $75 a week, to be cross-trained for three 
months as an educational researcher, seemed extraordinary and inappropriate 
to us, given the original objectives of the program. 

It is now apparent that programs of this sort need to consider the possibility 
of ''maximum qualifications" as well as the more obvious "minimum qualifica- 
tions ." Perhaps the basic guiding thought when this issue arose in this 
program was that it was a program designed for persons who were interested 
in educational R&D, but who did not already possess the natural entry keys. 

It was a program designed to provide opportunities to persons with basic 
credentials, who might not otherwise have had those opportunities, to enter 
educational research and development. Individuals with Ph.D.'s in any 
field or individuals who had already been heavily involved in educational 
R&D seemed suspect. The general thinking was that these persons ought 
to be able to obtain entry into the field without the program. We also 
had the rather common thoughts that persons who had been super high-powered 



in their former positions would somehow be dissatisfied with the junior 
positions likely to result from the program, and that this disparity would 
lead to dissatisfaction, frustration, and perhaps, short job tenure. 

From the viewpoint of the unemployed, this agrument has often been heard 
and just as often criticized. These are, perhaps, judgments which reviewers 
of applications should not make. If the applicant is willing, then perhaps 
he should be given the opportunity. These diverse views are obviously not 
easily resolved. Reasons for rejection depend on why the particular applicant 
audience was targeted, and no general statements can be made. Suffice to 
say that upper limits as well as lower limits need to be considered. What- 
ever those limits are depends on the reasons for tapping that particular 
population in the first place. 



Additional reasons for applicant rejection 



Other reasons for applicant rejection by the judges included clear non- 
association with aerospace, already employed, very little reference to 
education, consistent and persistent job-jumper, no degree, extraordinary 
salary expectations, and poor academic or employment record. 

While this pool no doubt included a number of persons who would have made 
substantial contribution to the program, and smooth adaptation to education- 
al R&D, the decision was made to hold to the advertised requirements, out 
of fairness to all. This turned out to be the wisest course of action. 

While a number of the unqualified seemed otherwise very attractive, con- 
stantly making individual exceptions can lead to very muddy waters, indeed. 

A program needs to define its basic selection criteria at the very beginning, 
and faithfully adhere to those criteria . 



M. In addition to how might trmmees be selected , who might be 
selected^ in future re^Haatians 0/ the model ? 

The notion of cross-training in this model is basically sound. The program 
certainly worked to a more than reasonable degree, and the manpower pool 
seemed strongly responsive to a program of this type. But there are some 
serious considerations as to whether or not the pool needed to have the 
basic characteristic of being unemployed. It is felt that this single 
characteristic contributed to a great deal more real and imagined "static" 
than might otherwise have been the case. 

It is important to .note that selection for the program depended, in part, 
on genuine motivation to join the program. A program wants trainees who 
are maximally interested in educational R&D work and who are thus more 
likely to have maximum success. The factor of being unemployed raises ob- 
vious questions of doubt as to whether the decision to pursue program 
membership was sincerely based in interest or whether it was based in just 
pursuing another channel to find some opportunity, in some field, just for 
the sake of potential employment. The model itself has no relationship 
to this question, nor did the criterion of being a physical scientist or 
aerospace worker have anything to do with this question. It is totally 
related to that one selection factor of unemployment. 

There are, of course, program purposes and philosophies which would call 
for far less concern with this issue of genuine interest. This would be 
the case for programs which were initiated because of social or economic 
conscience. However, if the issues are ease and speed of transition into 
educational R&D, new and fresh ideas into educational R&D, or confidence 
of interest in educational R&D, then the factor of unemployment becomes much 
more a point of confusion than substantive confirmation. Responsiveness 
in this case had to be tempered by the audience. 

Cross-training or retraining programs for unemployed aerospace workers, in 
any variety of areas, were very popular during 1971. Some were effective, 
many were not. The big unknown is whether or not participants were in 
the programs willingly and enthusiastically, or only because they were 
areas to which to turn in times of severe employment stress. This can 
never be genuinely answered. 

It may well be that future programs conducted for the benefit of strengthen- 
ing the area of educational R&D might well select from an audience who is 
gravitating toward education for more natural reasons . This is not to 
necessarily doubt those who entered this program under the specified con- 
ditions, but the degree of confidence in genuine interest and commitment 
to the field would have been enhanced had the target population been some- 
thing other than the relatively long-term unemployed. In summary, unem- 
ployment was felt to be a strong and, perhaps, unnecessary, intervening 
variable; and future programs might be targeted for more obvious audiences, 
no matter what model is employed. 



CHAPTER 4 

WHAT OTHER TYPES OF MODELS OR MODIFICATIONS 
OF THIS MODEL MIGHT BE CONSIDERED? 



During the course of the program, varieties of models for the development 
of educational research and development personnel were discussed and con- 
sidered. These models fell into three broad categories: (1) the Independent 

Training Model, (2) the "Adjunct" Model, and (3) the In-Service Model. 

Each will be discussed in turn in the following paragraphs. 



The Independent Training Model 



In the training model, there are two key variables, the audience being trained 
and the target level of the intended employment positions. Common to all 
of the audiences is that they are new to educational R&D and have had no 
special or formal contact with education. There are, of course, other 
variables which could also be considered, such as intensity of training effort 
and the length of the training program, but in terms of explaining the model , 
it will be assumed that these particular variables are held relatively 
constant across audiences in combination with intended sophistication of 
outcome. 

Firstly, there are many potential audiences for training, but three examples 
would adequately describe the major categories. The aerospace trainees of 
the present project reflect the highest degree of technical sophistication 
which trainees would normally be expected to bring to the training setting. 
These trainees were already highly educated, experienced workers who had had 
some involvement with research and development, but in a different field. 
Trainees of this type could include professionals from any number of occu- 
pational areas, not directly associated with educational R&D. Examples are 
sociologists, economists, demographers, advertising personnel, political 
scientists, teachers, writers, and, of course, physical scientists of nu- 
merous types. The main goal of training programs for such types of trainees 
would be to reapply that expertise to the problems and processes of educational 
research and development, i.e., to cross-train. The ultimate goal of place- 
ment would be breadth of experience and fairly rapid advancement to key 
positions. 

The second category of potential trainees are those who have collegiate 
education, but little work history, and almost no past contacts with edu- 
cational R&D. An example would be the new college graduate with a B.A. 
in history. The ultimate employment level is likely to be that of the 
professional after some years of experience in educational R&D. But the 



immediate targeted level of employment after a comparatively short 
training program would be that of an entry level position. 

The third group could be labeled "non-professional." These would include 
secretaries who are moving into research assistant positions or experienced 
teacher aides, housewives, etc. Here, the targeted levels of employment 
are likely to be of a paraprofessional entry level position. 

A training program for any of these three categories of the population is 
intended to be experiential, not exclusively academic. The issue would 
be to provide an orientation and realistic work experiences in educational 
R&D for persons who have had none and are not likely to have it otherwise. 
Obviously, training can be directed for many audiences and toward a variety 
of work outcomes, ranging all the way from entry level, clerk-type positions 
to key professionals. Which combination of factors a program wishes to 
emphasize in the training model depends on personnel needs in educational 
R&D, the desire for "new blood" in educational R&D, and even political, 
economic, and social realities. The training model of this study considered 
all three: there are predicted needs for experienced professionals, there 

is an acknowledged desire for expanded areas of expertise, and unemployed 
aerospace workers were a national problem which needed efforts at resolution. 

The employment of a training model can thus be focused in a variety of 
directions depending on intent and philosophy. In some cases, there may 
be a desire to accelerate the opportunities for the bright and new-to-the- 
work-world who have expressed interest in educational R&D. In other in- 
stances, programs may have a more altruistic purpose, such as assisting in 
employment opportunities for minority groups, whose opportunities have been 
stifled in the past. Some basic philosophical differences may arise here. 

At odds could be the question of whether a program is initiated to help 
society or help educational research and development. Hopefully, both goals 
might be met in the same program. However, whatever the intent or philosophy 
of a program, it will be reflected in the criteria for trainee selection 
and in the potential employment positions of the trainees. In any case, 
the apprenticeship training model is flexible, for it could work across 
all settings and audiences, within limits. One key qualifier is that the 
training setting needs to be equipped to provide a variety of work experi- 
ences. The second qualifier is that the training program not be seen as a 
school, in the academic sense. No work-oriented institution is prepared to 
meet academic needs. Training in this model needs to be offered in light 
of who ts being trained, what they are being trained for, and why they were 
selected for training in the first place. 

There are some basic weaknesses in this model. First, it could be somewhat 
difficult for the host institution to provide all of the preliminary know- 
ledges, work habits, and basic background in education desirable of even 
entry level personnel. There is neither the time nor the staff to offer 
such personalized assistance in areas which are somewhat peripheral to 
actual productivity in educational research and development. This po- 



tential problem, of course, would diminish as the sophistication and ex- 
perience of the trainee grew. It could certainly be the case that a host 
institution would want to take on the total, basic training of a totally 
inexperienced worker, but a training program of that type is more likely 
to focus on a philosophy of altruism than making practical and immediate 
inputs to the field of educational R&D. Thus, the training model is 
probably most easily employed when a minimum of guidance is to be offered 
each trainee. In addition it is clearly a training model, not a broad scope, 
multi-purpose educational program. 

In summary, the training model is flexible and can meet a variety of needs 
relevant to educational R&D. However, such a model needs strong definition 
as to population and intent and, in addition, needs some degree of confidence 
that training will, in fact, result in a reasonable number of satisfactory 
placements. 



The Adjunct Model 



An alternative to the training model has been labeled the "Adjunct Model"— 
adjunct in the sense that it is conducted in a parallel fashion with a 
formal study program at some university or college. This model would probably 
be somewhat more comfortable for the host institution in that there is not 
.he same degree of sensed responsibility for the trainee's future as there 
is in the independently run training model. In addition, the student would 
spend a portion of his time in traditional classwork and study in an academic 
institution and a portion of his time in a real-life work setting. Such 
arrangements have been labeled work-study programs, and more recent versions 
have been labeled "engineered internships." In the latter case, the student 
would theoretically spend a portion of his day or week in formal study and 
the remainder of his day or week in real-life application of that study. 

This would be a highly desirable arrangement, and its concept seems to be 
growing in popularity. However, in most instances, engineered internships 
are not realistic. Firstly, in basic philosophy, they are modeled on the old 
notion of "teacher training": the student teacher is placed in a real school 

under the sponsorship of a classroom teacher, but his ultimate evaluation 
still lies with the institution or professors at the university, college, or 
whatever. In this sense, the academic institution attempts to supervise, 
monitor, and evaluate the activities in the work setting. This is simply 
not a very practical approach. The organization providing the work experience 
cannot tailor make its activities for the benefit of the student or for what 
the student's academic institution perceives as his benefit. Work-study 
arrangements cannot be organized around the student unless the work is mani- 
festly artificial or unless the university is extremely flexible in the way 
it accumulates student "credit." Ordinarily the students' activities need 
to revolve around those of the institution. The student needs to mold him- 
self to the ebb or flow of the work of the organization: otherwise the work 
experience is not genuine. 
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In addition, faculty members representing the academic institution and super- 
vising the student frequently cannot define what are deemed worthwhile 
student activities in the various organizations proyiding the work experience. 
Such an attempt would no doubt be met with great resistance and a mild furor. 
It is not conceivable that a facul ty member visit such organizations as AIR, 
ETS, or the Par West Lab and try to prescribe the activities of the student. 

In the first place, it would be an intrusion; and in the second place, what 
the instructor thinks that students need might not be available at that time, 
or ever, in the organization. If it is not a natural and common activity, 
the organization would hardly be equipped to offer it. 

There are, perhaps, two conditions under which the "Adjunct Model," charac- 
terized by the engineered internship, might be workable. The first of these 
would be if the work portion were somewhat removed from the study portion, 
without an overly studied attempt to match work with academics. This would 
be typical of the highly successful and well established work-study program 
at Antioch. The organizations are expected to provide work, but not study, 
experiences. The student is expected to accomodate to the ebb and flow of 
whatever work he is in during the work portion of the program. The workday 
follows the pattern of a normal workday for a number of weeks or months. 

There is not the interrupted workday while students go back to clAss, so 
typical of other work-study programs. In this situation, a more real-work 
life experience is likely to be had, and the opportunity for more substantial 
contributions on the part of the student is likely to be enhanced. 

What is characteristic here is that the academic institution is willing to 
give up control of the student for intermittent periods while, at the same 
time, having confidence that the organization where the student is placed is, 
in fact, offering the student a valuable work experience. From the view- 
point of the organization, the student's presence places no great load on 
either staff time or the normal flow of production. It is an easy arrange- 
ment from both sides. 

Even if an academic institution is unwilling to give up control of the 
student or to place a degree of confidence in the organization spot soring 
che work, there is still one other set of conditions which could make the 
Adjunct Model practical; although economically, they are not likely. That 
would be if the academic institution were, in some way, willing to underwrite 
the activities of its students. If it is somehow expected that the work 
organizations will offer academic study or will tailor make its activities 
for the benefit of the. student, then some compensation to the organization 
needs to be offered. To offer academics, which is unnatural to the organi- 
zation, or to provide specialized experiences, which is uncommon to the 
organization, would result in extraordinary direct and indirect expenditures. 
Organizations might be willing to fulfill these expectations, but to do so 
without some sort of external underwriting is not reasonable. 

Therefore, while the adjunct model sounds like a good idea, it is difficult 
to carry off without some very special and mutually understood conditions. In 
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most cases, this model as it is currently approached, is not particularly 
attractive to potential host institutions. 



The In-Service Model 
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The third model is labeled the "In-Service Model,” and it is by far the most 
practical; and in terms of today's employment patterns in educational K&U, 
perhaps the most important of the three discussed. 

It is practical in the sense that there is little intrusion on the organi- 
zation. Trainees are easily selected and bring to the training situation 
some commitment to, and experience in, educational R&D, and presumably strong 
potential for expanded contribution to the field. The institutional focus 
for that contribution is already defined. 

The emphasis in the model would not be basic training, as in the other two 
models, but rather provision for expanded growth in the field of educational 
R&D based on a foundation of already acquired basic skills and experience. 

Trainees would be drawn from a wide variety of agencies, organizations, and 
institutions already involved, either directly or indirectly, in educational 
R&D. These would include profit or non-profit agencies, the various educa- 
tional centers and laboratories, public school systems, federal and state 
departments of education, and the like. Personnel from these various settings 
could range all the way from paraprofessionals to senior professors. Inde- 
pendent institutions are perhaps more accountability oriented than subsidized 
institutions engaged in educational R&D. There is much to be learned from 
environments of the former in their sense of urgency and mission, their need 
for competitive creativity, and the general necessity for the management of 
products and personnel in a businesslike setting. 

The basic manpower problem in educational R&D today is not how to get more 
persons into the field, but how to upgrade the personnel we already have. 

There need to be provisions for providing the strongest possible experiences, 
and a variety of experiences, to as many of the existing personnel in the 
field as possible. Such a provision could allow for the strengthening of the 
field as a whole. These are times, perhaps, when we do not need "more bodies," 
but "better bodies." We need to capitalize on the partially developed ex- 
pertise which is already there, and an In-Service Training Program seems 
the most fruitful direction. 

This will become progressively even more important given the initiation and 
development of the National Institute for Education. The design for NIE 
calls for a great deal of eventual in-house activity. NIE is going to make 
tremendous demands on experienced manpower, particularly in the areas of 
planning, research, and evaluation. A substantial portion of this demand 
might be met through an effective In-Service Training Program. 
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As in any training model, there is still the problem of crediting or certi- 
fication. However, there are new movements which might be used to overcome 
this traditional problem. Programs of the In-Service type might begin to 
tie into the newly emerging "University Without Walls," and external degree 
programs such as are being developed in New Jersey, New York, Minnesota, 
and California. 

It would be highly appropriate for the field of educational R&D to further 
explore in-service training and crediting programs of this type. Current 
possibilities in this general direction are offered by the programs at the 
Minnesota Metropolitan State College where "graduation" is based on certified 
competency; and in the traditionally conservative New York State Board of 
Regents, Edison College is an exclusively external study college being 
developed in cooperation with the Carnegie Foundation in the state of New 
Jersey. 

With some attempts at organization, contacts, and development, there is no 
reason why In-Service Programs sponsored by private institutions could not 
offer certification through these emerging advanced study structures. 

It is an exciting idea and could offer a more potent influx of experienced, 
productive workers to the field of educational research and development 
than has ever before been realized. 



APPENDIX A 



TRAINEE SURVEY 



As part of our final studies of the training program, we are interested in 
your responses to the following questions. Your comments may by typed or 
written; it is not necessary that you identify yourself. Please give 
your comments to Jim, Gene, or Judy by late Tuesday afternoon, or early 
Wednesday morning, at the latest. Thank you for your help. 

1) To what extent do you feel you know more about educational R&D in 
general, than you did thirteen weeks ago? 

2) In which specific topic areas did you gain the most information from 
the program? 

3) What specific skills related to educational activities were additionally 
or solely developed because of your participation in the program? 

4) To what extent did you pursue outside, independent study on topics of 
interest, topics related to apprenticeships, or topics which you thought 
would be useful in educational R&D? 

5) Which components of the program seemed to be most useful in your acqui- 
sition of information and/or skills, i.e., seminars, independent readings 
and contacts, informal discussions, apprenticeship activities, and so 
forth? 

6) Would you have desired more or less structure in terms of readings, formal 
seminars, or other types of directed study? 

7) Would you have desired more mandatory activities, deadlines, etc.? 

8) Do you feel you could or should have started on apprenticeships the 
first few days of the program? 

9) How would you compare the effectiveness of a) one or two long term 
apprenticeships versus b) a series of short term apprenticeship 
experiences? 

10) What recommendations would you make for future programs? 

11) To what extent do you feel you are more prepared to enter the field of 
educational R&D now, than you were thirteen weeks ago? 
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